Closed tadam98s closed 4 months ago
You need to pass bypass_robots=True to the get_config function. The error states that your local website has robots.txt rule which prohibits bot or script access. It can be just bypassed using the arguments above.
config = get_config(url,bypass_robots=True) wp = config.create_page() wp.get(config['project_url']) form = wp.get_forms()[0]
Exception has occurred: IndexError
list index out of range
File "D:\download\test\scripts\clone_test.py", line 21, in
You need to verify whether their are forms before applying [0] index. Common sense yaar. Check the url property of the wp object before hand whether their wasn't any redirects. Then check the available forms using get_forms method.
print(wp.url) http://localhost:9000
when I open the site manually I get:
if I login manually, can I manually copy the cookies and pass to pywebsave? Apparently, there is a java script that shows the login/password form. Is there a way to provide it with the answers programatically? or continue after manual login ?
If is only showing the spinner and the java code /js/outBIMYN2XL.js that opens the login/password does not appear to be executed.
<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8" charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<link rel="apple-touch-icon" href="/apple-touch-icon.png">
<link rel="apple-touch-icon" sizes="57x57" href="/apple-touch-icon-57x57.png">
<link rel="apple-touch-icon" sizes="60x60" href="/apple-touch-icon-60x60.png">
<link rel="apple-touch-icon" sizes="72x72" href="/apple-touch-icon-72x72.png">
<link rel="apple-touch-icon" sizes="76x76" href="/apple-touch-icon-76x76.png">
<link rel="apple-touch-icon" sizes="114x114" href="/apple-touch-icon-114x114.png">
<link rel="apple-touch-icon" sizes="120x120" href="/apple-touch-icon-120x120.png">
<link rel="apple-touch-icon" sizes="144x144" href="/apple-touch-icon-144x144.png">
<link rel="apple-touch-icon" sizes="152x152" href="/apple-touch-icon-152x152.png">
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon-180x180.png">
<link rel="icon" type="image/x-icon" href="/favicon.ico">
<meta name="application-name" content="test" />
<meta name="msapplication-TileColor" content="#FFFFFF" />
<meta name="msapplication-TileImage" content="/mstile-512x512.png" />
<title>test</title>
<link rel="stylesheet" href="/js/outWHCP76XN.css" />
</head>
<body>
<div id="content" data-base-url="" data-server-status="UP" data-instance="test" data-official="true">
<div class="global-loading">
<i class="spinner global-loading-spinner"></i>
<span aria-live="polite" class="global-loading-text">Loading...</span>
</div>
</div>
<script type="module" src="/js/outBIMYN2XL.js"></script>
</body>
</html>
Just login with your browser and then copy the cookies to the pywebcopy session headers.
kindly show me where these session headers are. -- Mickey Cohen Shanit Ltd. CEO POB 23410, Jerusalem 9123302, Israel M: +972-54-758-6312 Skype: tadam_98 @.***
On April 5, 2024 2:51:53 PM GMT+03:00, Raja Tomar @.***> wrote:
Just login with your browser and then copy the cookies to the pywebcopy session headers.
-- Reply to this email directly or view it on GitHub: https://github.com/rajatomar788/pywebcopy/issues/124#issuecomment-2039605812 You are receiving this because you authored the thread.
Message ID: @.***>
You can access the session using the .session attribute of the wp object that you created. Then use .headers attribute of the session to set the headers including cookies. The session object is a requests library session. You can read up online how to manage a requests.Session object.
URLmain = "http://localhost:9000/" session = requests.session() my_cookies = {'JWT-SESSION': 'some value', 'XSRF-TOKEN': 'some value'} r = requests.post(URLmain, cookies=my_cookies)
This sets the cookies correctly. But the next save_website is not tied to this session at it gets the url as a parameter not the session. How do I connect this session to the following save_website ?
Use the wp object style approch as you did in the start. Use wp.get methods to open pages. Then the session would remain same for all the requests.
I have built a url that sets cookies.
url_cookies = f"{url}/cookies/set?JWT-SESSiON={my_cookies['JWT-SESSiON']}?XSRF-TOKEN={my_cookies['XSRF-TOKEN']}"
Then I used get_config to start a session
config = get_config(url, project_folder, project_name=projectName, bypass_robots=True, debug=True, delay=None, threaded=False)
Then I was not sure how to use the usl_cookies to set the cookies. I tried:
crawler = config.create_crawler()
crawler.get(url_cookies)
crawler.get(url)
crawler.save_complete(pop=True)
But this did not set the cookies. Not sure how to use the wp as if I have the cookies not sure I need to open a form. Please avise.
Anyway, I maybe copying the cookies may not work as they are JWT and could be associated with some seed in each instance. I may be back to the question, how to login.
You may have proceed with trial and error method. It is understood that there is no javascript support in the pywebcopy. So each javascript based site would require some different approach to get around. At the moment I can only tell you to see the requests.Session usage and documentation. Because cookies and auth is handled by that quite capable library.
Hi,
I have a website running on a docker that is accesses locally: http://localhost:9000/dashboard?id=face-animation
I need to login with two fields: login and password.
When I run it I get on wp.get(config['project_url']):
How do I write the code to save this website ? When I login to the site it creates two cookies: JWT-SESSION XSRF-TOKEN Which I need to carry on into the pywebsave