dat-ecosystem-archive / datBase

Open data sharing powered by Dat [ DEPRECATED - More info on active projects and modules at https://dat-ecosystem.org/ ]
http://datbase.org
244 stars 32 forks source link

Sub-directory listings only display one file #636

Open da2x opened 5 years ago

da2x commented 5 years ago

All of these directories contains files that datBase doesn’t display in directory listings. You can navigate to the files directly (/poster.jpeg will work for all the below examples) and datBase will show the file.

https://datbase.org/dat://0f9840c371387d372148484fb622038758a197781723b5e1d034e7f6f4daf643/contents/instapaper-scrolling-android https://datbase.org/dat://0f9840c371387d372148484fb622038758a197781723b5e1d034e7f6f4daf643/contents/microsoft-todo-scrolling-android https://datbase.org/dat://0f9840c371387d372148484fb622038758a197781723b5e1d034e7f6f4daf643/contents/spideroak-one-checkbox

evdubs commented 5 years ago

I am experiencing this issue as well. Here are two repositories:

In EPS and Sales Estimates, eps-estimate/2019-02-17.csv is visible, but none of the other files are. However, the other files are still accessible. 2019-02-10.csv is an example of a file that is available to download but is not visible on datbase.org.

I discussed this with martinheidegger on IRC and we went over making sure the files were being properly hosted by dat share. martinheidegger posted the following:

Hmm, its kind-of tricky to pinpoint the error here. As far as I read https://github.com/mafintosh/hypercore-archiver/blob/master/index.js#L253-L260 is where the replication magic happens. All cores are sparse: https://github.com/mafintosh/hypercore-archiver/blob/master/index.js#L362 - This means that the file tree also will be replicated sparsely. (it requests the files as the user requests files.

Is someone able to take a look at what's going on here?

da2x commented 5 years ago

Another affected repository: https://datbase.org/dat://e71784fbaaac13e7a1b20ba39d8149b2ba893171204672575eab8b4fd0758ec3/contents/1382

evdubs commented 5 years ago

Also, using http://ranger.mauve.moe/dat-js-example, we can see the repository does have more than one file in a subdirectory.

da2x commented 5 years ago

OK, after some digging I found the problem. Directory listings only show files added in the latest revision that added any files to the directory. It doesn't show files that were already there. You can see this here: https://datbase.org/dat://e71784fbaaac13e7a1b20ba39d8149b2ba893171204672575eab8b4fd0758ec3/contents/1388

okdistribute commented 5 years ago

Would like to have some discussions with @RangerMauve on how we can strip down datbase's interface (take the css/design) and plug in dat-js-example behind so we can remove these bugs. Datbase's code has gotten a bit stale at this point and could use a refresh. Thanks for finding these things!

RangerMauve commented 5 years ago

Yeah, I think we could get pretty far with the stuff I've been playing around with in load-dat-page