Missing thumbnails in merged mode

[MasterPetrik]

IMPORTANT BUG!

In last version I very often see a problem with randomly missing images in results grid.

• With every repeating search missing images are different.
• This empty space is not clickable.
• Please pay attention to the images results counter. It shows real number of the images, but some results are missed for some reason.
• Moreover, if you open neighbor to the missing image image in preview window and then go to the missing image in preview window, it will be correctly loaded. In preview window you can see all result including all images, missing in the results grid.
• Looks like it some king of GUI problem with results grid.

First page loaded

only 9 of 22 images shown

Second page loaded

only 20 of 44 images shown after loading second page images grid remakes and now missing images are different compare with the previous screenshot, even on the first page part of the grid! Some missing images are shown now, and some shown images are missing now...

Second page loaded (another trying from different tab)

only 31 of 44 images shown Another trying shows that same actions leads to different results, another images are missing from first try.

Third page loaded

only 23 of 47 images shown after loading third page images grid remakes again here is the result

How often does this problem occur? On which sources?

Randomly but very often, only in 5.5.0 (in previous versions all work fine).

Used merged results + endless page mode, all sources settings are default

[Bionus]

Is there anything showing up in the log when this happens? Do you have any blacklist set that might cause this?

[MasterPetrik]

Do you have any blacklist set that might cause this? No. And how can you explain why missing images are always different? That's obviously not connected to it.

And I checked, shown and missing images in results in the grid can be from same source, so I guess this is not on server side, considering that in preview all images(even missing in grid) are shown correctly

Is there anything showing up in the log when this happens?

Just tried another search, loaded first page:

and here is the log text for that page

log text.txt

for my view, nothing special in log found

[Bionus]

Can't reproduce unfortunately. :cry:

Do the missing images change if you resize the window around?

and here is the log text for that page log text.txt for my view, nothing special in log found

Indeed, nothing to see there.

[Bionus]

Ok my bad I can reproduce. It seems to only happen when using yande.re's big thumbnails: removing it from the selected sources "fixed" the grid display for me.

[MasterPetrik]

It seems to only happen when using yande.re's big thumbnails: removing it from the selected sources "fixed" the grid display for me.

Indeed. Removing it from the selected sources "fixed" the grid display for me too.

But, tell the truth, grid filling system is made pretty bad, I guess it's need a fully rework: overlaying images, missing images, wrong images order in grid, repainting all grid instead of adding items to the end of grid while loading next page in endless page mode There are so many problems in it, and most of them appears randomly and not easy to force reproduce.

Obviously grid system is not optimal. Rework at least may fix these problems even without finding it's origin(like with overlaying problem), I guess.

[ghost]

I thought I would really like the grid view, but due to my slow internet, the loading was also slow so I found I liked the paginated view better. In my short time using it, however, I didn't see any problems (probably because I only use one or two sources).

[Bionus]

Yes, the current implementation of the fixed-width grid layout has some weird behaviors.

That's because I had to re-implement the whole layout system manually: Qt does not provide built-in this kind of layout. It only provides some basic layouts (horizontal, vertical, grid, etc.).

Also, some problems you mention don't come from the layout itself. Overlaying images and wrong images order in grid for example are not related. What's related is the missing images (maybe), issue #955, or the weird flashy stuff (redraws, etc.) occurring when adding new images.

I'll see if some improvements can be done on this end. Another option would be to use the "official" flow layout, which while not integrated in Qt, still is supported by them. The problem with this layout is the alignment of the items: They're all sticked together, and if there is too much space, you'll get a margin on the right side:

One could split this margin between the left and right side (by "centering" the whole layout), but the current Grabber layout splits it between every images, which is better in my opinion.

[Bionus]

I thought I would really like the grid view, but due to my slow internet, the loading was also slow so I found I liked the paginated view better. In my short time using it, however, I didn't see any problems (probably because I only use one or two sources).

You mean the fixed-width grid layout? Why would your slow internet make it more ugly/slow? The only difference is the number of images per line: the old layout uses sqrt(images.count()) while the new one somehow uses window width / 150.

[Bionus]

Well I went "screw it" and re-coded it all. The layout should be smooth now, while keeping all its alignment stuff. Maybe it will fix #955 too (not sure).

[ghost]

@bionus, after reading what you said, I think my slow internet hunch is false. I was referring to the redraw lag stuff which sounds like it was just the way it was. I wish I knew QT, so I could contribute to stuff like this, as I hate seeing so many people report stuff but not really contributing actual code. Seems like it would be annoying to you.

[Bionus]

@> after reading what you said, I think my slow internet hunch is false. I was referring to the redraw lag stuff which sounds like it was just the way it was

So this should hopefully be fixed in the next release (or in the develop branch thanks to commit 95714753b55ba482bf57db0300f56ffcc19209fa. Those on Windows who want to try it out and confirm it's indeed behaving better can find it in the nightly version (there's an issue I'm aware of where the scroll area gets a big useless white space at the bottom).

I wish I knew QT, so I could contribute to stuff like this, as I hate seeing so many people report stuff but not really contributing actual code. Seems like it would be annoying to you.

Yeah, I also wish more people contributed with code, but I can't really expect all the users to be able to contribute to the project, especially since it's not that easy to bootstrap and get into (even though this was greatly improved recently).

Those giving suggestions, finding bugs and reporting them in a polite/clear manner are also being helpful by helping me fix the bugs more easily.

And when I feel overwhelmed by the number of issues, or annoyed by the behavior of some users, I just take a break from the project and come back when I'm itching to code something (which is what I was doing until last week). But to be honest, recently, more and more tickets are for suggestions, or small, "easy to fix" bugs, which is quite nice too.

[ghost]

@Bionus I'm excited to check out the new grid!

[ghost]

Looks good to me, but if

is turned off, it starts stacking the images. In infinite scrolling, it seems to load and stack indefinitely (if the above check box is not checked). Really nice though if scroll area is enabled! I like it. The previous issues are fixed from what I can tell.

Also, scrolling seems a bit laggy, but that's probably QT's fault. I found someone talking about it here: https://stackoverflow.com/questions/33046694/qt-workaround-for-slow-vertical-scrolling-in-qgraphicsview-with-many-items

Edit: Not sure if your infinite loop commit solves one of the problems mentioned above, but that just goes to show I should hold up reporting anything if it's in the dev branch.

[MasterPetrik]

Also, some problems you mention don't come from the layout itself. Overlaying images and wrong images order in grid for example are not related. What's related is the missing images (maybe), issue #955, or the weird flashy stuff (redraws, etc.) occurring when adding new images.

I don't mean that they are related. But all these problems are result of some problems with grid filling, right?

That's because I had to re-implement the whole layout system manually

the current Grabber layout splits it between every images, which is better in my opinion.

Indeed. And implementation is good enough. Better enchance it than use "official" flow layout. I'm sorry for my words that overestimated some problems,

Well I went "screw it" and re-coded it all. The layout should be smooth now, while keeping all its alignment stuff. Maybe it will fix #955 too (not sure).

955 looks fixed indeed, but do not rush and close that issue for now. I will make some checks in the nearest future to be sure.

[MasterPetrik]

Those on Windows who want to try it out and confirm it's indeed behaving better can find it in the nightly version (there's an issue I'm aware of where the scroll area gets a big useless white space at the bottom).

I'm using last nightly version and can report that new grid system doesn't fixed problem with missing images because of yande.re. They are still missing in randow amount each reload of results.

First try of search, only 20\21 images shown

Third retry of search, 20\21 images shown(image which missing in the grid is opened in preview window):

After ~5 rerty of search in same tab you finally can see all results in the grid:

Also you can see a strange behavior of grabber with results for this request "kuze_matsuri" . Thile there are only 21 original images, and all of them from page 1 from each source. But grabber says that there are also page 2 of resuts (you can see "more results" button on the screenshot above) But when you press it, nothing will change, same number of results! Not a singe new image!

Moreover, if you just go to page 2 by ">" button then you will see that there is 0 images on the page 2! but page 2 is exist anyway! Something looks not right, maybe bug?

If you wanna try to reproduce it, here is my sources(https on gelbooru is enabled) :

But now at least there are no more holes in new grid from missing images. (maybe overlaying images and other displacement issues now fixed too, who knows) Anyway that's a big step in improving grid system, so keep it up, you are on the right way!

P.C. And please fix big useless white space at the bottom if you can, it's not very comfort to use grabber with it(

Good luck!

[Bionus]

I'm using last nightly version and can report that new grid system doesn't fixed problem with missing images because of yande.re. They are still missing in randow amount each reload of results.

That's actually a whole new problem with the merged results. I'll work on it, thanks for letting me know.

Also you can see a strange behavior of grabber with results for this request "kuze_matsuri" . Thile there are only 21 original images, and all of them from page 1 from each source. But grabber says that there are also page 2 of resuts (you can see "more results" button on the screenshot above) But when you press it, nothing will change, same number of results! Not a singe new image!

Moreover, if you just go to page 2 by ">" button then you will see that there is 0 images on the page 2! but page 2 is exist anyway! Something looks not right, maybe bug?

I can't really reproduce this issue currently. Maybe the image setup changed? (the second page actually contains images for me)

But this can happen if for example all images on page 2 are already present on page 1 from other sources. The fact that manually loading the second page doesn't show anything is weird however.

And please fix big useless white space at the bottom if you can, it's not very comfort to use grabber with it

Yes it's planned before the official release. 😉

[Bionus]

Ok actually the issue has never been related to yande.re. It was merely a coincidence.

The root cause was the modification done to implement #1025: when replacing an image with the new one with better tags, if the thumbnail of the first image hadn't loaded yet, it will be "lost", causing the blank spaces.

The fix should be included in the latest nightly. :+1:

[MasterPetrik]

The fix should be included in the latest nightly.

Tested new Nightly a lot. Here is my report: Indeed, now problem with missing thumblails is solved for 98% cases. Only one search request is still causing missing of the only one same thumbnail for some reason. All other tested requests with different parameters showed fully correct grid with right number of items even for several endless-scroll-pages. So it's a great sucsess. Thanks for actualy rework of grid filling system, now the new one is much stable compare to old one. Also I don't faced with old bugs with grid in new grid yet, so I can guess that they won't appear in new grid(But it's still too early for closing them, need more usage for clear result). Also I don't see a new bugs/problems in new nightly, so I think, it's almost ready to release.

Also problem with increases blank spaces in endless-page mode was fixed, thanks.

But the problem with blank space at the bottom of every results grid(even empty) in all (merged\unmerged) (paging\scrolling) modes is still relevant:

Hope you will fix it too.

And one more thing about new grid filling system.

In old system, at first was shown correct total number of images results, and after that grid filling started. In new system, I noticed, that at first grid filling starts, and only after that total number of images increases to correct number after a some seconds delay. Is that OK? Because now, for example, I can see 67 images, but total images number will be still 23 and only after some seconds it will show 67.

Anyway good job, looks like the main problem is solved and only few minor problems remains.

[Bionus]

Only one search request is still causing missing of the only one same thumbnail for some reason.

Which one?

Also I don't faced with old bugs with grid in new grid yet, so I can guess that they won't appear in new grid(But it's still too early for closing them, need more usage for clear result).

If you still don't see anything (I didn't meet any bug with it until now), feel free to close them whenever.

Also I don't see a new bugs/problems in new nightly, so I think, it's almost ready to release.

Well I pushed a lot of changes in the past few days, especially regarding the HTML APIs, so that might not be true anymore. :sweat: I'm planning a release this weekend though.

But the problem with blank space at the bottom of every results grid(even empty) in all (merged\unmerged) (paging\scrolling) modes is still relevant:

Hope you will fix it too.

In old system, at first was shown correct total number of images results, and after that grid filling started. In new system, I noticed, that at first grid filling starts, and only after that total number of images increases to correct number after a some seconds delay. Is that OK? Because now, for example, I can see 67 images, but total images number will be still 23 and only after some seconds it will show 67.

I just pushed a fix for those, it should have reached nightly. :+1:

[MasterPetrik]

Well I pushed a lot of changes in the past few days, especially regarding the HTML APIs, so that might not be true anymore.

For testing those changes, just using nightly Grabber is enough? or I need to download some other files from develop branch?

I'm planning a release this weekend though.

Hmm, maybe it's not to late to make a couple suggestions before release :thinking:

I just pushed a fix for those, it should have reached nightly.

Can confirm fixing those bugs, grid works very good and with no problems now.

Only one search request is still causing missing of the only one same thumbnail for some reason.

Which one?

Also, it's still relevant in the last nightly version too.

Search with "kuze_matsuri" request

But to reproduce it you should use the same source set(as it set on the screenshot) As you can see, only 20 from 21 images are shown(two pages loaded in scroll mode), repeating search alvays gives the same result. As you can see, image(opened in preview) is missing only in grid, but can be viewed in preview normally from the results list. That's strange.

Even more stange, that if you will use non merged mode, then you will see this image in grids from many sources, but if you change mode to merged, THIS image misses!

[Bionus]

For testing those changes, just using nightly Grabber is enough? or I need to download some other files from develop branch?

You'll have to update your model.xml files from the sites folder: https://github.com/Bionus/imgbrd-grabber/tree/develop/release/sites

Most have been updated with the new regular expressions, and new tag APIs (but those are not 100% tested yet).

Hmm, maybe it's not to late to make a couple suggestions before release 🤔

Well there's still 3~4 days left after all.

Can confirm fixing those bugs, grid works very good and with no problems now.

🎉

Search with "kuze_matsuri" request

Still can't reproduce, weird:

Can you share the log with the "-d" flag passed to Grabber in the latest nightly? (using a shortcut with the path "c:\path\to\Grabber.exe" -d should do the trick and show some more lines in the log)

Also, I see in your screenshots that even though you have sankaku and danbooru selected, they yield no results, maybe that might be linked? (mine give results) Do you have the list of your selected sources, excluding those who don't give any results?

[MasterPetrik]

Used sites pressets from develope branch

Searched with this set of sources(it's important):

Got different results in different tries with same settings:

Bad try

log bad.txt

Good try

log good.txt

[MasterPetrik]

After replacing all my sources with develop branch sources... Why now all my my sources icons looks like this? It tells that some update is aviable but I can't see where I can update them :thinking:

Also, let's count it as GUI bug: sources window have constant heigth, and with not many sources it's too much blank space in it. Tried resize the window, but changes doesn't saves(

[MasterPetrik]

Looks like the fix doesn't fixed updater after all

[Bionus]

I could reproduce indeed, it comes from a corrupted file from Safebooru: http://safebooru.org/thumbnails/1274/thumbnail_3da04833a454153e885384bfb9886dda681773a4.jpg

Given this kind of issue is not widespread, very specific etc, from now on Grabber will show a placeholder images for failed thumbnail loads (it won't try other sources in merged mode, the cost/gain is too high).

[MasterPetrik]

Given this kind of issue is not widespread, very specific etc, from now on Grabber will show a placeholder images for failed thumbnail loads (it won't try other sources in merged mode, the cost/gain is too high).

Congatulations with succesful investigation! Indeed, now it's works correct. Now If thumbnail is broken, you resize original image and make thimbnail from it, am I right?

[MasterPetrik]

Also I found another strange behavior, if you mind to investigate it too...:smirk:

Same sources set. Last nightly.

"8c" search request

For some reason I don't have a "load more results" button(endless page button mode enabled)at the bottom of the window in both merged and non merged modes, also I can go to the next results page via ">" button, and it will work fine, but it's very-very strange, for other search requests all work fine.

[MasterPetrik]

also sometimes many images in results grid are not clickable, I don't know the reason, but it happens only in new grid system

[Bionus]

Because of some other issue that I also fixed last hour.

[MasterPetrik]

Ok, all mantioned here issues indeed has beed fixed in 5.5.1. I suppose new issues with grid I will post in new issues, thanks