search

kiwix / kiwix-apple

Kiwix for iOS & macOS
https://apple.kiwix.org
GNU Lesser General Public License v3.0
487 stars 70 forks source link

There seems to be no protection against fetching again an again the same version of the catalog #1007

Open kelson42 opened 1 week ago

kelson42 commented 1 week ago

If I click twice on "fetch catalog" it takes many seconds each time to complete. Which tends to let me think there is no check if the version of the online catalogue is the same as the copy we have locally. If the version would be checked and proven to be the same (via Etag), then Kiwix should not redownload everything and it should be over in 0.1s.

See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match

kelson42 commented 1 week ago

@BPerlakiH Can you please confirm/explain how it works?

BPerlakiH commented 5 days ago

@kelson42 I can see in the traffic that the catalog is refetched on each button press: https://library.kiwix.org/catalog/v2/entries?count=-1 It seems to have the correct headers though:

Cache-Control: max-age=0, must-revalidate
ETag: "1726580544242481858.2213904/z"

The amount of time it takes is divided though. From what I can see the request (on a good wifi) takes only 900ms, whereas parsing and (maybe inserting the data) takes longer, around 2 more seconds.

The libkiwix library is also attempting to download the favicons, which I think is not really needed as it just simply not working, so we download those "on demand", when displaying the list of categories. It is also creating a lot of log outputs:

Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/da7fb796-e4ee-641f-41ff-6d7da49a28cb/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/4f489a41-d2dd-8066-7136-d21277420e45/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/f73302be-60bc-b740-53ba-50b1624eb4b9/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/e2c51a75-6700-cb90-69bd-f22a9de13482/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/646f6c7f-60da-6a29-ebf6-47384265edce/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/c9470bd8-571d-4815-76d3-fb4086036613/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/9f6dcc7e-a1dc-4868-4a8f-30d01a38ba29/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/2460effe-5812-9c25-1cfa-1ea824d9a843/?size=48
Cannot download favicon from https://library.kiwix.org/catalog/v2/illustration/fca187d6-425a-2e11-7867-eb61e2a7916e/?size=48
....

over 3200 lines of those.

kelson42 commented 5 days ago

@kelson42 I can see in the traffic that the catalog is refetched on each button press:

Thx for confirming the bug. Please read documentation I have given and fix both HTTP request and response handling.

kelson42 commented 5 days ago

The libkiwix library is also attempting to download the favicons, which I think is not really needed

If you believe the libkiwix does not work properly, please open an issue there.