Feature request: Update downloads

[nisehime]

Every time I want to update my downloads (download new works from an artist) I have to restart them. The issue comes with sites like Twitter, where some users have thousands of files. When you restarting, it grabs all the links again, so it takes a lot of time to update the whole download list, also it makes redundant traffic. It's also problematic when you have cleaned up the folder from the garbage and want to download new works, but the program redownloads all files again.

So I'm asking for update option for the downloaded entries. I'm not sure how it'd be better to implement. You can either make a database which stores all downloaded links/IDs from every entry, so when you choose to update it downloads everything, but those links (or it can store only the ID of the last downloaded file). Or simply download new works up to the latest file in the folder, without database. (but it probably would work only if the filename contains the ID)

[KurtBestor]

Try this: https://github.com/KurtBestor/Hitomi-Downloader/releases/tag/Technical-Preview

[nisehime]

@KurtBestor I don't see anything.

UPD: It seems that you just broke the restart button. This is not what I mean. Don't change the behavior of restarting download. I'm asking for a new action in the context menu: "Update". It should work on the stage of collecting links (before download starts)

[KurtBestor]

What's wrong?

[nisehime]

@KurtBestor As I understand, your program has 2 basic stages of downloading. On the first stage it searches for the direct links to the images (see pic1) and then on the second stage it downloads all them. When you click "Restart (S)", Hitomi begins collecting all the links again. I want you to make an action "Update" near the Restart. This action should search only for the new links and then download them.

For example, when I click Restart on an entry with 5000 files, I have to wait when Hitomi finds all the 5000 (+new) links again and then it downloads only missing (new) files. But when I click "Update" it should collect only new links, and once it reaches to the last downloaded one, the "Reading..." stage should stop and then start downloading.

pic1


[KurtBestor]

Yes, Now Restart will work as you described.

Restart acts like "Update" for completed tasks.

[nisehime]

@KurtBestor It doesn't. When I clicked restart on your Technical Preview it still were "Reading..." through ALL 5000 links. Yes, it didn't re-download previous files, though, but I need reduced "Reading..." stage time too.

OK, I've downloaded latests Preview and now it seems to work. But still, assigning this function to the Restart button seems like a bad idea. This is confusing. What if I actually want to restart it, redownload missing files? Removing a task from the list and adding it again or marking as incomplete kinda sucks and may not be obvious to some people. Also some entries I have are in permanent "incompleted" state because some links are 404 or something. I know I can mark them as completed, but I don't really want to. I want to know it was incompleted and what files are missing.

[KurtBestor]

Actually, Restart was introduced to perform following functions:

• "Retry" failed tasks
• "Update" completed tasks (most likely social media)

It would be more confusing if there were Restart and Update at the same time.

You can check the missing files in Task info; Right click on the task and press Info... (I). Missing files are marked in red.

[nisehime]

I did some tests, seems like it's working fine. I found a problem with the page counter, it doesn't update when the latests file is removed, so it'd be better to store the last ID separately I guess, though it's not critical. But I've also found it doesn't work for this artist: https://twitter.com/ashiomi_masato and few others. It just restarts normally, like before, not sure why, they're in completed state.

Does it only work for Twitter? Pixiv, pawoo doesn't seem to support this feature, but it'd be nice to see this on other sites too.

You can check the missing files in Task info; Right click on the task and press Info... (I).

The error are removed from the log once you mark a task as completed and restart the program.

[KurtBestor]

• Yes, it works for Twitter only. Aren't Pawoo & Pixiv fast enough?
• The error messages & URLs used to download are removed after restart the program if the task is successfully completed to improve memory consumption.

[nisehime]

Aren't Pawoo & Pixiv fast enough?

Well, Pixiv is faster, but not that much, and wouldn't say Pawoo is. It's just less common and artists here usually don't have 5K+ files. Anyway, I don't think it would be bad to add this feature to some other social network sites. At least over some time if it bothers you.

Could you do something with this:

But I've also found it doesn't work for this artist: https://twitter.com/ashiomi_masato and few others. It just restarts normally, like before, not sure why, they're in completed state.

I have some tasks in the list that don't use this update feature and they just restart completely every time. There seems to be nothing useful in the log so I won't post it (unless you ask). You probably need to check by yourself.

[nisehime]

Ok, removing a task from the list and adding it again solves the issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.

[nisehime]

Doesn't really work

[jack652]

wouldn't it be better to keep the restart button as it was ? the restart button should scan everything from the link and re-download anything that Hitomi finds missing in the downloads folder and add a separate update button that only looks for new Files that weren't there before

[rickmiron]

I support the motion, with all violence