Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.5k stars 216 forks source link

Use automatic download of favorites and monitoring. Twitter image download is incomplete. #2400

Closed kosserYN closed 8 months ago

kosserYN commented 3 years ago

Sometimes pictures older than one month cannot be downloaded Sometimes unable to get any pictures Sometimes is incomplete Use modify lastCheck

monitors.json

`{

        "cumulated": 26,
        "delay": 0,
        "download": true,
        "filenameOverride": "amrkdrw/amrkdrw-%id%-%date:format=yyyyMMdd_mszz%-img%md5%.%ext%",
        "getBlacklisted": false,
        "interval": 36000,
        "lastCheck": "2020-01-21T16:09:23Z",
        "notify": false,
        "pathOverride": "D:/OneDrive/Documents/2d",
        "postFilters": [
        ],
        "preciseCumulated": true,
        "query": {
            "tags": [
                "Username",
                "amrkdrw",
                "retweets:{no}"
            ]
        },
        "sites": [
            "api.twitter.com"
        ]
    },

LOG

[00:40:04.660][Info] Monitoring tick [00:40:04.660][Info] Monitoring new images for 'Username amrkdrw retweets:{no}' on 'Twitter' [00:40:04.662][Info] [api.twitter.com][Json] Loading page https://api.twitter.com/1.1/statuses/user_timeline.json?count=20&include_rts=false&exclude_replies=false&tweet_mode=extended&screen_name=amrkdrw [00:40:04.698][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.361][Info] [api.twitter.com][Json] Receiving page https://api.twitter.com/1.1/statuses/user_timeline.json?count=20&include_rts=false&exclude_replies=false&tweet_mode=extended&screen_name=amrkdrw [00:40:05.365][Info] [api.twitter.com][Json] Parsed page https://api.twitter.com/1.1/statuses/user_timeline.json?count=20&include_rts=false&exclude_replies=false&tweet_mode=extended&screen_name=amrkdrw: 9 images (9), 0 tags (0), -1 total (-1), -1 pages (-1) [00:40:05.366][Info] Added 3 images to the download queue for monitor 'Username amrkdrw retweets:{no}' on 'Twitter' [00:40:05.367][Info] Not enough information to directly load the image (from blacklist: 1 / from file url: 0 / from filename tags: 0/0) [00:40:05.368][Info] Loading and saving image from https://pbs.twimg.com/media/ExdPcWeVcAElgBU.jpg:orig in D:\OneDrive\Documents\2d\amrkdrw\amrkdrw-1375655565053939713-20210327 465900-img.jpg [00:40:05.369][Info] Not enough information to directly load the image (from blacklist: 1 / from file url: 0 / from filename tags: 0/0) [00:40:05.369][Info] Loading and saving image from https://pbs.twimg.com/media/ExPvYc4XEAAnBmO.jpg:orig in D:\OneDrive\Documents\2d\amrkdrw\amrkdrw-1374705522809597957-20210324 515200-img.jpg [00:40:05.370][Info] Not enough information to directly load the image (from blacklist: 1 / from file url: 0 / from filename tags: 0/0) [00:40:05.371][Info] Loading and saving image from https://pbs.twimg.com/media/EwLKmarVkAENMBp.jpg:orig in D:\OneDrive\Documents\2d\amrkdrw\amrkdrw-1369880015794364417-20210311 17100-img.jpg [00:40:05.402][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.406][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.410][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.963][Info] Monitoring tick [00:40:05.964][Info] Next monitoring will be in 9869 seconds

Many thanks

Bionus commented 3 years ago

From those messages, are you using some VPN or proxy that is playing around with your networking settings while Grabber is performing the loads?

[00:40:05.402][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.406][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.410][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID.

PS: why are you using retweets:{no}? It should be retweets:no.

kosserYN commented 3 years ago

Yes, I can’t link to twitter without proxy

From those messages, are you using some VPN or proxy that is playing around with your networking settings while Grabber is performing the loads?

I found that monitors will only download the first page of sources images per page 20 and will not download tweets with multiple images This seems to be related to the number of retweets and text tweets from the artist

few retweets and tweets only one page

2021-06-30

fewer retweet and tweet can be downloaded to the set time image

An Artists with a lot of retweets and tweets only one page not download tweets with multiple images

2021-06-30 (2)

An artist with a lot of retweets and tweets Nothing will download image

redstoneleo commented 2 years ago

From those messages, are you using some VPN or proxy that is playing around with your networking settings while Grabber is performing the loads?

[00:40:05.402][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.406][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID. [00:40:05.410][Warning] [Qt][qt.network.monitor] Could not get the INetworkConnection instance for the adapter GUID.

PS: why are you using retweets:{no}? It should be retweets:no.

Yes, any idea on solving the problem ?

stale[bot] commented 9 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

If this issue is about a bug that still happens in the latest version, or a suggestion that is still relevant, feel free to comment on it and the maintainers will have another look, they might have missed it!

Thank you!