Closed lucacham closed 11 months ago
Environment
button, then the Copy
button, and paste the copied text here).Thank you for your reply Below is my version info.
SCrawler v2023.8.27.0 (x64) OS: Microsoft Windows 11 Pro (x64) ffmpeg version: ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers yt-dlp version: 2023.07.06 gallery-dl version: 1.25.8 cURL version: curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/3.0.7 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 WinIDN libssh2/1.10.0 nghttp2/1.50.0 ngtcp2/0.10.0 nghttp3/0.7.1 libgsasl/2.2.0
I don't know what and how you configured GDL through python. SCrawler doesn't need special GDL configurations.
Because I saw in the help documentation that cookies and gallery-dl are required, I installed gallery-dl.
Because I saw in the help documentation that cookies and gallery-dl are required, I installed gallery-dl.
It is, but SCrawler handles it on its own. SCrawler doesn't use user configurations. SCrawler creates and uses its own GDL configuration. It is possible that your GDL configuration conflicts with SCrawler's GDL configuration. I think you should remove your GDL configuration and everything will be fine.
As shown below, I have deleted the gallery-dl I configured myself and restarted my computer
I re-downloaded SCrawler, re-imported the cookies, and then tried to download Twitter pictures. The result was the same as before: a new Twitter account folder was created during the download. Under the folder was a Settings folder, and the settings folder contained xml. file
Can't confirm. I can download this profile https://twitter.com/hajisan0525 Show me a screenshot of your environment settings.
The same cannot be done now, but the log is as follows
2023-08-31 22:39:09: [Twitter] - tuzaibaby
: user information saving error
路径中具有非法字符。
2023-08-31 22:39:09: Message: 路径中具有非法字符。
HelpLink:
StackTrace: 在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:39:38: [Twitter] - xixi8848
: user information saving error
路径中具有非法字符。
2023-08-31 22:39:38: Message: 路径中具有非法字符。
HelpLink:
StackTrace: 在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:39:44: gallery-dl configuration file creating error
不支持给定路径的格式。
2023-08-31 22:39:44: Message: 不支持给定路径的格式。
HelpLink:
StackTrace: 在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
在 SCrawler.API.Twitter.UserData.GdlCreateConf(SFile Path)
2023-08-31 22:39:44: [Twitter] - tuzaibaby
: [Twitter] - tuzaibaby
: GetTimelineFromGalleryDL()
不支持给定路径的格式。
2023-08-31 22:39:44: Message: 不支持给定路径的格式。
HelpLink:
StackTrace: 在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
在 SCrawler.API.Twitter.UserData.GetTimelineFromGalleryDL(CacheKeeper Cache, CancellationToken Token)
2023-08-31 22:39:44: [Twitter] - tuzaibaby
: user information saving error
路径中具有非法字符。
2023-08-31 22:39:44: Message: 路径中具有非法字符。
HelpLink:
StackTrace: 在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:40:38: gallery-dl configuration file creating error
不支持给定路径的格式。
2023-08-31 22:40:38: Message: 不支持给定路径的格式。
HelpLink:
StackTrace: 在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
在 SCrawler.API.Twitter.UserData.GdlCreateConf(SFile Path)
2023-08-31 22:40:38: [Twitter] - tuzaibaby
: [Twitter] - tuzaibaby
: GetTimelineFromGalleryDL()
不支持给定路径的格式。
2023-08-31 22:40:38: Message: 不支持给定路径的格式。
HelpLink:
StackTrace: 在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
在 SCrawler.API.Twitter.UserData.GetTimelineFromGalleryDL(CacheKeeper Cache, CancellationToken Token)
environment
SCrawler v2023.8.6.0 (x64)
OS: Microsoft Windows 11 家庭中文版 (x64)
ffmpeg version: ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
yt-dlp version: 2023.07.06
gallery-dl version: 1.25.8
cURL version: curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/3.0.7 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 WinIDN libssh2/1.10.0 nghttp2/1.50.0 ngtcp2/0.10.0 nghttp3/0.7.1 libgsasl/2.2.0
@NormanMises Google translate: 路径中具有非法字符 - Illegal characters in path 不支持给定路径的格式 - The format of the given path is not supported
Try to specify the path in Latin characters
my path is E:\SCrawler\Twitter\
The following is a screenshot of my environment settings. (I have used downloading from Twitter, reddit, and xvideos. Except that Twitter cannot download pictures, etc., I can download from reddit and xvideos) I haven't changed these settings
In addition, the pictures below are my settings. Could you please help me see if there is anything wrong with my settings? Thank you.
![11](https://github.com/AAndyProgram/SCrawler/assets/25399625/23acec08-2547-4b58-8013-2b52432ddd1e) ![1](https://github.com/AAndyProgram/SCrawler/assets/25399625/6e553d5c-1861-4364-927f-f6130c52f8fb) ![2](https://github.com/AAndyProgram/SCrawler/assets/25399625/7ca8b3ac-db89-448a-80e6-cef0c6b925fe) ![3](https://github.com/AAndyProgram/SCrawler/assets/25399625/dbf6b91d-fb66-48d2-816a-64958789fe2f) ![4](https://github.com/AAndyProgram/SCrawler/assets/25399625/daaa0e68-a950-41a7-953d-df14cc1dadcc) ![5](https://github.com/AAndyProgram/SCrawler/assets/25399625/6eac03b8-d88e-468a-95bd-817e567fcb3a) ![7](https://github.com/AAndyProgram/SCrawler/assets/25399625/1f35dce7-2031-4d79-a634-6e6c21f577a4) ![8](https://github.com/AAndyProgram/SCrawler/assets/25399625/68dde024-c2d9-43ce-921b-b8c7f790299d) ![9](https://github.com/AAndyProgram/SCrawler/assets/25399625/ed431ecd-82f9-4aec-8fbc-43082875baf1) ![10](https://github.com/AAndyProgram/SCrawler/assets/25399625/847d0ed1-e6ec-4b5b-b800-7e26f2d9becc)
I used to use a chrome plugin before but since Elon restricted its API, the pluning doesn't do anything, and I tried to download from this program but it doesn't work, I hit start download and it ends abruptly.
![Uploading image.png…]()
The following is a screenshot of my environment settings. (I have used downloading from Twitter, reddit, and xvideos. Except that Twitter cannot download pictures, etc., I can download from reddit and xvideos) I haven't changed these settings
try downloading new version, i doit and works.
@lucacham @NormanMises Try this script and tell me if any files are downloaded
"D:\\OutputFolder\\"
)<SCRAWLER_SETTINGS_PATH>
text with the actual SCrawler settings path<URL>
text with the actual URL to the user profile<PATH_TO_GDL>
test with the actual path to the gallery-dlcd /d "D:\\OutputFolder\\"
<PATH_TO_GDL>\gallery-dl --no-download --no-skip --cookies "<SCRAWLER_SETTINGS_PATH>\Responser_Twitter_Cookies_Netscape.txt" --write-pages <URL> -D "D:\\OutputFolder\\"
@Alexo88 Good to know that everything is fine.
Unfortunately this script didn't help me, nothing downloads :(
This script should download information in JSON files. If there is nothing there, so I think you've configured cookies incorrectly and/or something wrong with the Japanese location in GDL. Sorry. I have no issues with Twitter. Everything works for me.
Strange. I can't get anything from three Twitter users...
Unable to download, even if a proxy is used
D:\Apps\SCrawler\Environment\gallery-dl --no-download --no-skip --cookies "D:\Apps\SCrawler\Settings\Responser_Twitter_Cookies_Netscape.txt" --write-pages https://twitter.com/kkkl6587 -D "E:\SCrawler\test\" --proxy "http://127.0.0.1:12345"
[twitter][error] HttpError: SSLError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: /i/api/graphql/XA6F1nJELYg65hxOC2Ekmg/UserByScreenName?variables=%7B%22screen_name%22%3A%22kkkl6587%22%2C%22withSafetyModeUserFields%22%3Atrue%7D&features=%7B%22hidden_profile_likes_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))
Have you tested without a proxy? No luck?
I have configured gallery-dl through python(I'm sure gallery-dl is installed successfully and I can download Twitter's pictures by executing the command), and exported the Netscape file through the browser extension Get cookies.txt LOCALLY, and imported the cookie into SCrawler, but after clicking download (after a few seconds, it prompts that the download is complete), there is only one Settings in the download folder folder without any image files. I don't know what is wrong with the settings.. No log showing any errors
↓ there is only one Settings in the download folder folder without any image files![20230830154534](https://github.com/AAndyProgram/SCrawler/assets/25399625/4673e3c2-4430-4b85-a4e3-9764f46fabf8)
Files in the settings folder![20230830160947](https://github.com/AAndyProgram/SCrawler/assets/25399625/d44b5027-dbc2-4df9-b74f-82a8ca8f046b)