Closed pulsar70 closed 5 years ago
Provide some used code or log details.
config - INFO - Got response <Response [200]> from http://www.writeopinions.com/robots.txt
config - INFO - Got response <Response [200]> from http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri
webpage - INFO - Starting save_complete Action on url: ['http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri']
parsers - INFO - Parsing tree with source: <<urllib3.response.HTTPResponse object at 0x11AF7E90>> encoding
The log output stops there when in a thread.
In a self-standing python interpreter:
kwargs = {'project_name' : 'dummy'} pywebcopy.save_webpage(url='http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri', project_folder='c:\work', **kwargs) config - INFO - Got response <Response [200]> from http://www.writeopinions.com/robots.txt config - INFO - Got response <Response [200]> from http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri webpage - INFO - Starting save_complete Action on url: ['http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri'] parsers - INFO - Parsing tree with source: <<urllib3.response.HTTPResponse object at 0x045AD270>> encoding
and parser <<lxml.etree.HTMLParser object at 0x044609F0>> webpage - INFO - Starting save_assets Action on url: 'http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri' webpage - Level 100 - Queueing download of <29> asset files. webpage - INFO - Starting save_html Action on url: 'http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri' webpage - INFO - WebPage saved successfully to c:\work\dummy\www.writeopinions.com\c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.html config - INFO - Got response <Response [200]> from http://t3.gstatic.com/images?q=tbn:ANd9GcRNmVn169BsI2COBefR47BN6eDbuKZ18B4AXdm-5un12uldcw4d6k9dwc0 elements - INFO - Writing file at location c:\work\dummy\t3.gstatic.com\11d6e711images.jpg elements - INFO - File of type .jpg written successfully to c:\work\dummy\t3.gstatic.com\11d6e711images.jpg config - INFO - Got response <Response [200]> from http://t0.gstatic.com/images?q=tbn:ANd9GcQEWTHxP9VC7soNKgEZpIegGh8zvZV1h1ABtQR5Gpt8V85Kqfx8ITUVv3A elements - INFO - Writing file at location c:\work\dummy\t0.gstatic.com\9b67b38aimages.jpg elements - INFO - File of type .jpg written successfully to c:\work\dummy\t0.gstatic.com\9b67b38aimages.jpg config - INFO - Got response <Response [200]> from http://t3.gstatic.com/images?q=tbn:ANd9GcRH8DNu8Q0BHGrdQW2vWtFRhWmhAurISHkI1LrA3MnalPVkPJ-FU97HHOw config - INFO - Got response <Response [200]> from http://t2.gstatic.com/images?q=tbn:ANd9GcQmJSdsNcOqsYnCA8qmUtwhfvmzo-VC6-jow3XF3Hq2EyW4N1vstbrNG_D6 elements - INFO - Writing file at location c:\work\dummy\t3.gstatic.com\084a7c15images.jpg config - INFO - Got response <Response [200]> from http://www.writeopinions.com/logo.png elements - INFO - Writing file at location c:\work\dummy\t2.gstatic.com\9a3a25efimages.jpg elements - INFO - Writing file at location c:\work\dummy\www.writeopinions.com\a0cc724dlogo.png elements - INFO - File of type .png written successfully to c:\work\dummy\www.writeopinions.com\a0cc724dlogo.png elements - INFO - File of type .jpg written successfully to c:\work\dummy\t3.gstatic.com\084a7c15images.jpg elements - INFO - File of type .jpg written successfully to c:\work\dummy\t2.gstatic.com\9a3a25efimages.jpg config - INFO - Got response <Response [301]> from http://www.google.com/trends/embed.js?hl=en-US&q=abu%20abd%20al%20rahman%20ibn%20aqil%20al%20zahiri&content=1&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=330 config - INFO - Got response <Response [200]> from http://connect.facebook.net/en_US/all.js#xfbml=1 elements - INFO - Writing file at location c:\work\dummy\connect.facebook.net\en_US\2beb0397all.js config - INFO - Got response <Response [200]> from http://www.writeopinions.com/es.gif elements - INFO - Writing file at location c:\work\dummy\www.writeopinions.com\f1643c95es.gif config - INFO - Got response <Response [200]> from http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri elements - INFO - File of type .gif written successfully to c:\work\dummy\www.writeopinions.com\f1643c95__es.gif elements - INFO - File of type .js written successfully to c:\work\dummy\connect.facebook.net\en_US\2beb0397all.js config - INFO - Got response <Response [200]> from https://trends.google.com/trends/embed.js?hl=en-US&q=abu%20abd%20al%20rahman%20ibn%20aqil%20al%20zahiri&content=1&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=330 elements - INFO - Writing file at location c:\work\dummy\www.google.com\trends\c98a0889embed.js elements - INFO - File of type .js written successfully to c:\work\dummy\www.google.com\trends\c98a0889embed.js elements - INFO - Writing file at location c:\work\dummy\www.writeopinions.com\c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.pwc elements - INFO - File of type .htm written successfully to c:\work\dummy\www.writeopinions.com\c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.pwc core - INFO - Saved the Project as ZIP archive at c:\work\dummy.zip core - INFO - Downloaded Contents Size :: 23 KB's
Typical case of irresponsible server request.
Try a process instead of thread. or deamon. Or Use WebPage class method which have the collection of threads at ._thread attribute, you can manually join them.
On Sat, Jun 22, 2019, 10:42 PM pulsar70 notifications@github.com wrote:
config - INFO - Got response <Response [200]> from http://www.writeopinions.com/robots.txt config - INFO - Got response <Response [200]> from http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri webpage - INFO - Starting save_complete Action on url: [' http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri'] parsers - INFO - Parsing tree with source: <<urllib3.response.HTTPResponse object at 0x11AF7E90>> encoding and parser <<lxml.etree.HTMLParser object at 0x11AEF030>> webpage - INFO - Starting save_assets Action on url: ' http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri' webpage - Level 100 - Queueing download of <29> asset files. webpage - INFO - Starting save_html Action on url: ' http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri' webpage - INFO - WebPage saved successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.writeopinions.com \c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.html config - INFO - Got response <Response [200]> from http://t0.gstatic.com/images?q=tbn:ANd9GcQEWTHxP9VC7soNKgEZpIegGh8zvZV1h1ABtQR5Gpt8V85Kqfx8ITUVv3A elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t0.gstatic.com \9b67b38a__images.jpg elements - INFO - File of type .jpg written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t0.gstatic.com \9b67b38aimages.jpg config - INFO - Got response <Response [200]> from http://www.writeopinions.com/logo.png elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.writeopinions.com \a0cc724dlogo.png config - INFO - Got response <Response [200]> from http://t2.gstatic.com/images?q=tbn:ANd9GcQmJSdsNcOqsYnCA8qmUtwhfvmzo-VC6-jow3XF3Hq2EyW4N1vstbrNG_D6 elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t2.gstatic.com \9a3a25efimages.jpg elements - INFO - File of type .png written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.writeopinions.com \a0cc724dlogo.png elements - INFO - File of type .jpg written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t2.gstatic.com \9a3a25ef__images.jpg config - INFO - Got response <Response [200]> from http://connect.facebook.net/en_US/all.js#xfbml=1 elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\connect.facebook.net \en_US\2beb0397all.js config - INFO - Got response <Response [200]> from http://t3.gstatic.com/images?q=tbn:ANd9GcRNmVn169BsI2COBefR47BN6eDbuKZ18B4AXdm-5un12uldcw4d6k9dwc0 elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t3.gstatic.com \11d6e711images.jpg elements - INFO - File of type .jpg written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t3.gstatic.com \11d6e711__images.jpg config - INFO - Got response <Response [200]> from http://t3.gstatic.com/images?q=tbn:ANd9GcRH8DNu8Q0BHGrdQW2vWtFRhWmhAurISHkI1LrA3MnalPVkPJ-FU97HHOw elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t3.gstatic.com \084a7c15images.jpg elements - INFO - File of type .jpg written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\t3.gstatic.com \084a7c15images.jpg config - INFO - Got response <Response [301]> from http://www.google.com/trends/embed.js?hl=en-US&q=abu%20abd%20al%20rahman%20ibn%20aqil%20al%20zahiri&content=1&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=330 config - INFO - Got response <Response [200]> from http://www.writeopinions.com/es.gif elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.writeopinions.com \f1643c95__es.gif config - INFO - Got response <Response [200]> from http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.writeopinions.com \c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.pwc elements - INFO - File of type .gif written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.writeopinions.com \f1643c95es.gif elements - INFO - File of type .htm written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.writeopinions.com \c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.pwc elements - INFO - File of type .js written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\connect.facebook.net \en_US\2beb0397all.js config - INFO - Got response <Response [200]> from https://trends.google.com/trends/embed.js?hl=en-US&q=abu%20abd%20al%20rahman%20ibn%20aqil%20al%20zahiri&content=1&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=330 elements - INFO - Writing file at location C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.google.com \trends\c98a0889embed.js elements - INFO - File of type .js written successfully to C:\work\www_ZNZRAC_ABC_1to500\URLs_cache\ZN_101772\www.google.com \trends\c98a0889__embed.js
The log output stops there when in a thread.
In a self-standing python interpreter:
kwargs = {'project_name' : 'dummy'} pywebcopy.save_webpage(url=' http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri', project_folder='c:\work', **kwargs) config - INFO - Got response <Response [200]> from http://www.writeopinions.com/robots.txt config - INFO - Got response <Response [200]> from http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri webpage - INFO - Starting save_complete Action on url: [' http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri'] parsers - INFO - Parsing tree with source: <<urllib3.response.HTTPResponse object at 0x045AD270>> encoding and parser <<lxml.etree.HTMLParser object at 0x044609F0>> webpage - INFO - Starting save_assets Action on url: ' http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri' webpage - Level 100 - Queueing download of <29> asset files. webpage - INFO - Starting save_html Action on url: ' http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri' webpage - INFO - WebPage saved successfully to c:\work\dummy\ www.writeopinions.com\c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.html config - INFO - Got response <Response [200]> from http://t3.gstatic.com/images?q=tbn:ANd9GcRNmVn169BsI2COBefR47BN6eDbuKZ18B4AXdm-5un12uldcw4d6k9dwc0 elements - INFO - Writing file at location c:\work\dummy\t3.gstatic.com \11d6e711images.jpg elements - INFO - File of type .jpg written successfully to c:\work\dummy\ t3.gstatic.com\11d6e711images.jpg config - INFO - Got response <Response [200]> from http://t0.gstatic.com/images?q=tbn:ANd9GcQEWTHxP9VC7soNKgEZpIegGh8zvZV1h1ABtQR5Gpt8V85Kqfx8ITUVv3A elements - INFO - Writing file at location c:\work\dummy\t0.gstatic.com \9b67b38aimages.jpg elements - INFO - File of type .jpg written successfully to c:\work\dummy\ t0.gstatic.com\9b67b38aimages.jpg config - INFO - Got response <Response [200]> from http://t3.gstatic.com/images?q=tbn:ANd9GcRH8DNu8Q0BHGrdQW2vWtFRhWmhAurISHkI1LrA3MnalPVkPJ-FU97HHOw config - INFO - Got response <Response [200]> from http://t2.gstatic.com/images?q=tbn:ANd9GcQmJSdsNcOqsYnCA8qmUtwhfvmzo-VC6-jow3XF3Hq2EyW4N1vstbrNG_D6 elements - INFO - Writing file at location c:\work\dummy\t3.gstatic.com \084a7c15images.jpg config - INFO - Got response <Response [200]> from http://www.writeopinions.com/logo.png elements - INFO - Writing file at location c:\work\dummy\t2.gstatic.com \9a3a25efimages.jpg elements - INFO - Writing file at location c:\work\dummy\ www.writeopinions.com\a0cc724dlogo.png elements - INFO - File of type .png written successfully to c:\work\dummy\ www.writeopinions.com\a0cc724dlogo.png elements - INFO - File of type .jpg written successfully to c:\work\dummy\ t3.gstatic.com\084a7c15images.jpg elements - INFO - File of type .jpg written successfully to c:\work\dummy\ t2.gstatic.com\9a3a25efimages.jpg config - INFO - Got response <Response [301]> from http://www.google.com/trends/embed.js?hl=en-US&q=abu%20abd%20al%20rahman%20ibn%20aqil%20al%20zahiri&content=1&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=330 config - INFO - Got response <Response [200]> from http://connect.facebook.net/en_US/all.js#xfbml=1 elements - INFO - Writing file at location c:\work\dummy\ connect.facebook.net\en_US\2beb0397all.js config - INFO - Got response <Response [200]> from http://www.writeopinions.com/es.gif elements - INFO - Writing file at location c:\work\dummy\ www.writeopinions.com\f1643c95es.gif config - INFO - Got response <Response [200]> from http://www.writeopinions.com/abu-abd-al-rahman-ibn-aqil-al-zahiri elements - INFO - File of type .gif written successfully to c:\work\dummy\ www.writeopinions.com\f1643c95__es.gif elements - INFO - File of type .js written successfully to c:\work\dummy\ connect.facebook.net\en_US\2beb0397all.js config - INFO - Got response <Response [200]> from https://trends.google.com/trends/embed.js?hl=en-US&q=abu%20abd%20al%20rahman%20ibn%20aqil%20al%20zahiri&content=1&cid=TIMESERIES_GRAPH_0&export=5&w=500&h=330 elements - INFO - Writing file at location c:\work\dummy\www.google.com \trends\c98a0889embed.js elements - INFO - File of type .js written successfully to c:\work\dummy\ www.google.com\trends\c98a0889embed.js elements - INFO - Writing file at location c:\work\dummy\ www.writeopinions.com\c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.pwc elements - INFO - File of type .htm written successfully to c:\work\dummy\ www.writeopinions.com\c69aeed5abu-abd-al-rahman-ibn-aqil-al-zahiri.pwc core - INFO - Saved the Project as ZIP archive at c:\work\dummy.zip core - INFO - Downloaded Contents Size :: 23 KB's
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rajatomar788/pywebcopy/issues/15?email_source=notifications&email_token=AIGSNTSMF5D5TL25JDMRXCDP3ZMOTA5CNFSM4H2WYUQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYKNS7Q#issuecomment-504682878, or mute the thread https://github.com/notifications/unsubscribe-auth/AIGSNTT66B7F3P2LXQLZPUTP3ZMOTANCNFSM4H2WYUQA .
On Windows 10, executing: kwargs = {'project_name' : my_name} pywebcopy.save_webpage(url=my_url, project_folder=my_folder, **kwargs) works well in a self-standing python interpreter launched from a console.
Executing the exact same code from an active thread in a multi-threaded python code shows the call to save_webpage to never return.