search

folbricht / desync

Alternative casync implementation
BSD 3-Clause "New" or "Revised" License
343 stars 44 forks source link

When using multiple stores, it doesn't try 2nd store if chunk missing on first #238

Closed garyvdm closed 1 year ago

garyvdm commented 1 year ago

We have 2 stores that have eventual consistency, but there are periods of time when one of them may be missing chunks.

We thought that we could use desync to try both stores when fetching, but this seems to not be working.

Running build the latest version from master: 

~/desync/cmd/desync (git)-[master] # git describe
v0.9.3-8-g4d48f59

When chunk is only available on 2nd server, desync cache fails:

~/desync/cmd/desync (git)-[master] # ./desync cache -c /srv/prod_images/store.castr -s http://server1/store.castr/ -s http://server2/store.castr/ /srv/prod_images/archive/23-05-3/workstation.caibx
[>-----------------------------------------------------------------------------------------------------------------------------------------------------]   0.02% 0s
Error: http://server1/store.castr/: object c7f6/c7f666d66a77b2e37cc349327607c2192d8ecab521ab1ae5f432d5c9d19b23da missing from store

~/desync/cmd/desync (git)-[master] # curl --head http://server2/store.castr/c7f6/c7f666d66a77b2e37cc349327607c2192d8ecab521ab1ae5f432d5c9d19b23da
HTTP/1.1 404 Not Found
Server: webfs/1.21
Connection: Keep-Alive
Accept-Ranges: bytes
Content-Type: text/plain
Content-Length: 28
Date: Fri, 23 Jun 2023 13:18:34 GMT

~/desync/cmd/desync (git)-[master] # curl --head http://server2/store.castr/c7f6/c7f666d66a77b2e37cc349327607c2192d8ecab521ab1ae5f432d5c9d19b23da
HTTP/1.1 200 OK
Server: webfs/1.21
Connection: Keep-Alive
Accept-Ranges: bytes
Content-Type: text/plain
Content-Length: 42399
Last-Modified: Fri, 02 Jun 2023 08:13:28 GMT
Date: Fri, 23 Jun 2023 13:18:37 GMT
folbricht commented 1 year ago

Hmm, that looks like a bug. Would you be able to try this with local stores? That would help narrow it down, to see if it's only an issue with multiple HTTP stores.

folbricht commented 1 year ago

Pretty sure I found the bug and fixed it in https://github.com/folbricht/desync/pull/239 if you want to give this branch a try. It was only affecting HTTP stores, and must have been there for a while. Thanks for the detailed report.

ljjacobs commented 1 year ago

colleague of @garyvdm, tested #239 and it seems to be working as expected now.