search

AAndyProgram / SCrawler

🏳️‍🌈 Media downloader from any sites, including Twitter, Reddit, Instagram, Threads, Facebook, OnlyFans, YouTube, Pinterest, PornHub, XHamster, XVIDEOS, ThisVid etc.
https://discord.gg/uFNUXvFFmg
GNU General Public License v3.0
1.21k stars 81 forks source link

Can't download any pictures etc from twitter #164

Closed lucacham closed 11 months ago

lucacham commented 11 months ago

I have configured gallery-dl through python(I'm sure gallery-dl is installed successfully and I can download Twitter's pictures by executing the command), and exported the Netscape file through the browser extension Get cookies.txt LOCALLY, and imported the cookie into SCrawler, but after clicking download (after a few seconds, it prompts that the download is complete), there is only one Settings in the download folder folder without any image files. I don't know what is wrong with the settings.. No log showing any errors

20230830160705 20230830160721 20230830160743

↓ there is only one Settings in the download folder folder without any image files 20230830154534

Files in the settings folder 20230830160947

AAndyProgram commented 11 months ago
  1. The bug report is incomplete. Add version information (click the top right info button in the main window, then the Environment button, then the Copy button, and paste the copied text here).
  2. I don't know what and how you configured GDL through python. SCrawler doesn't need special GDL configurations.
lucacham commented 11 months ago

Thank you for your reply Below is my version info.

SCrawler v2023.8.27.0 (x64) OS: Microsoft Windows 11 Pro (x64) ffmpeg version: ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers yt-dlp version: 2023.07.06 gallery-dl version: 1.25.8 cURL version: curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/3.0.7 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 WinIDN libssh2/1.10.0 nghttp2/1.50.0 ngtcp2/0.10.0 nghttp3/0.7.1 libgsasl/2.2.0

I don't know what and how you configured GDL through python. SCrawler doesn't need special GDL configurations.

Because I saw in the help documentation that cookies and gallery-dl are required, I installed gallery-dl.

AAndyProgram commented 11 months ago

Because I saw in the help documentation that cookies and gallery-dl are required, I installed gallery-dl.

It is, but SCrawler handles it on its own. SCrawler doesn't use user configurations. SCrawler creates and uses its own GDL configuration. It is possible that your GDL configuration conflicts with SCrawler's GDL configuration. I think you should remove your GDL configuration and everything will be fine.

lucacham commented 11 months ago

As shown below, I have deleted the gallery-dl I configured myself and restarted my computer

20230831115501

I re-downloaded SCrawler, re-imported the cookies, and then tried to download Twitter pictures. The result was the same as before: a new Twitter account folder was created during the download. Under the folder was a Settings folder, and the settings folder contained xml. file

20230831135033 20230831135045

AAndyProgram commented 11 months ago

Can't confirm. I can download this profile https://twitter.com/hajisan0525 Show me a screenshot of your environment settings.

image

NormanMises commented 11 months ago

The same cannot be done now, but the log is as follows

2023-08-31 22:39:09: [Twitter] - tuzaibaby
    : user information saving error
    路径中具有非法字符。
2023-08-31 22:39:09: Message: 路径中具有非法字符。
    HelpLink:
    StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
    在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
    在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:39:38: [Twitter] - xixi8848
    : user information saving error
    路径中具有非法字符。
2023-08-31 22:39:38: Message: 路径中具有非法字符。
    HelpLink:
    StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
    在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
    在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:39:44: gallery-dl configuration file creating error
    不支持给定路径的格式。
2023-08-31 22:39:44: Message: 不支持给定路径的格式。
    HelpLink:
    StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
    在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
    在 SCrawler.API.Twitter.UserData.GdlCreateConf(SFile Path)
2023-08-31 22:39:44: [Twitter] - tuzaibaby
    : [Twitter] - tuzaibaby
    : GetTimelineFromGalleryDL()
    不支持给定路径的格式。
2023-08-31 22:39:44: Message: 不支持给定路径的格式。
    HelpLink:
    StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
    在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
    在 SCrawler.API.Twitter.UserData.GetTimelineFromGalleryDL(CacheKeeper Cache, CancellationToken Token)
2023-08-31 22:39:44: [Twitter] - tuzaibaby
    : user information saving error
    路径中具有非法字符。
2023-08-31 22:39:44: Message: 路径中具有非法字符。
    HelpLink:
    StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
    在 PersonalUtilities.Functions.XML.XmlFile.UpdateData(SFile _File, Boolean IgnoreVirtualDeclarations, ErrorsDescriber e)
    在 SCrawler.API.Base.UserDataBase.UpdateUserInformation()
2023-08-31 22:40:38: gallery-dl configuration file creating error
    不支持给定路径的格式。
2023-08-31 22:40:38: Message: 不支持给定路径的格式。
    HelpLink:
    StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
    在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
    在 SCrawler.API.Twitter.UserData.GdlCreateConf(SFile Path)
2023-08-31 22:40:38: [Twitter] - tuzaibaby
    : [Twitter] - tuzaibaby
    : GetTimelineFromGalleryDL()
    不支持给定路径的格式。
2023-08-31 22:40:38: Message: 不支持给定路径的格式。
    HelpLink:
    StackTrace:    在 PersonalUtilities.Functions.UniversalFunctions.ErrorsDescriber.Execute(ErrorsDescriber e, Exception ex, MMessage Message, Object OptionalValueReturn)
    在 PersonalUtilities.Functions.UniversalFunctions.SFile.Exists(SFO What, Boolean CreatePath, ErrorsDescriber e)
    在 SCrawler.API.Twitter.UserData.GetTimelineFromGalleryDL(CacheKeeper Cache, CancellationToken Token)

environment

SCrawler v2023.8.6.0 (x64)
OS: Microsoft Windows 11 家庭中文版 (x64)
ffmpeg version: ffmpeg version 5.1.2-full_build-www.gyan.dev Copyright (c) 2000-2022 the FFmpeg developers
yt-dlp version: 2023.07.06
gallery-dl version: 1.25.8
cURL version: curl 7.86.0 (x86_64-w64-mingw32) libcurl/7.86.0 OpenSSL/3.0.7 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.2 WinIDN libssh2/1.10.0 nghttp2/1.50.0 ngtcp2/0.10.0 nghttp3/0.7.1 libgsasl/2.2.0
AAndyProgram commented 11 months ago

@NormanMises Google translate: 路径中具有非法字符 - Illegal characters in path 不支持给定路径的格式 - The format of the given path is not supported

Try to specify the path in Latin characters

NormanMises commented 11 months ago

my path is E:\SCrawler\Twitter\ image

lucacham commented 11 months ago

The following is a screenshot of my environment settings. (I have used downloading from Twitter, reddit, and xvideos. Except that Twitter cannot download pictures, etc., I can download from reddit and xvideos) I haven't changed these settings

20230901090040

In addition, the pictures below are my settings. Could you please help me see if there is anything wrong with my settings? Thank you.

Screenshots 

![11](https://github.com/AAndyProgram/SCrawler/assets/25399625/23acec08-2547-4b58-8013-2b52432ddd1e)

![1](https://github.com/AAndyProgram/SCrawler/assets/25399625/6e553d5c-1861-4364-927f-f6130c52f8fb)

![2](https://github.com/AAndyProgram/SCrawler/assets/25399625/7ca8b3ac-db89-448a-80e6-cef0c6b925fe)

![3](https://github.com/AAndyProgram/SCrawler/assets/25399625/dbf6b91d-fb66-48d2-816a-64958789fe2f)

![4](https://github.com/AAndyProgram/SCrawler/assets/25399625/daaa0e68-a950-41a7-953d-df14cc1dadcc)

![5](https://github.com/AAndyProgram/SCrawler/assets/25399625/6eac03b8-d88e-468a-95bd-817e567fcb3a)

![7](https://github.com/AAndyProgram/SCrawler/assets/25399625/1f35dce7-2031-4d79-a634-6e6c21f577a4)

![8](https://github.com/AAndyProgram/SCrawler/assets/25399625/68dde024-c2d9-43ce-921b-b8c7f790299d)

![9](https://github.com/AAndyProgram/SCrawler/assets/25399625/ed431ecd-82f9-4aec-8fbc-43082875baf1)

![10](https://github.com/AAndyProgram/SCrawler/assets/25399625/847d0ed1-e6ec-4b5b-b800-7e26f2d9becc)
Alexo88 commented 11 months ago

I used to use a chrome plugin before but since Elon restricted its API, the pluning doesn't do anything, and I tried to download from this program but it doesn't work, I hit start download and it ends abruptly.

![Uploading image.png…]()

Alexo88 commented 11 months ago

The following is a screenshot of my environment settings. (I have used downloading from Twitter, reddit, and xvideos. Except that Twitter cannot download pictures, etc., I can download from reddit and xvideos) I haven't changed these settings

try downloading new version, i doit and works.

AAndyProgram commented 11 months ago

@lucacham @NormanMises Try this script and tell me if any files are downloaded

  1. Create a temporary directory (e.g. "D:\\OutputFolder\\")
  2. Replace the <SCRAWLER_SETTINGS_PATH> text with the actual SCrawler settings path
  3. Replace the <URL> text with the actual URL to the user profile
  4. Replace the <PATH_TO_GDL> test with the actual path to the gallery-dl
cd /d "D:\\OutputFolder\\"
<PATH_TO_GDL>\gallery-dl --no-download --no-skip --cookies "<SCRAWLER_SETTINGS_PATH>\Responser_Twitter_Cookies_Netscape.txt" --write-pages <URL> -D "D:\\OutputFolder\\"

@Alexo88 Good to know that everything is fine.

mykeehu commented 11 months ago

Unfortunately this script didn't help me, nothing downloads :(

AAndyProgram commented 11 months ago

This script should download information in JSON files. If there is nothing there, so I think you've configured cookies incorrectly and/or something wrong with the Japanese location in GDL. Sorry. I have no issues with Twitter. Everything works for me.

mykeehu commented 11 months ago

Strange. I can't get anything from three Twitter users...

NormanMises commented 11 months ago

Unable to download, even if a proxy is used

D:\Apps\SCrawler\Environment\gallery-dl --no-download --no-skip --cookies "D:\Apps\SCrawler\Settings\Responser_Twitter_Cookies_Netscape.txt" --write-pages https://twitter.com/kkkl6587 -D "E:\SCrawler\test\" --proxy "http://127.0.0.1:12345"
[twitter][error] HttpError: SSLError: HTTPSConnectionPool(host='twitter.com', port=443): Max retries exceeded with url: /i/api/graphql/XA6F1nJELYg65hxOC2Ekmg/UserByScreenName?variables=%7B%22screen_name%22%3A%22kkkl6587%22%2C%22withSafetyModeUserFields%22%3Atrue%7D&features=%7B%22hidden_profile_likes_enabled%22%3Afalse%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22subscriptions_verification_info_verified_since_enabled%22%3Atrue%2C%22highlights_tweets_tab_ui_enabled%22%3Atrue%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%7D (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1131)')))
AAndyProgram commented 11 months ago

Have you tested without a proxy? No luck?