search

MarginaliaSearch / PublicData

Public data sets for Marginalia Search
8 stars 2 forks source link

Document at what rate the public data sets are published #3

Open ncouture opened 4 months ago

ncouture commented 4 months ago

Hello MaginaliaSearch!

Upon finding this interesting open data sets, I noticed that their size is not very large and they already pre-date 2024 by a month or so.

Would you consider automating these exports? If not or if it requires a new issue in another project (MaginaliaSearch, for instance), could one document this in this repo's README.md?

Thank you and keep being awesome!

P.S. search.marginalia.nu is the search engine I've been wanting for years before it was created <3

vlofgren commented 4 months ago

Hey. These exports are currently a bit ad-hoc, I have some automation but it's not fully automated. They'll probably be updated at most one a quarter in the future, as the dataset isn't changing much faster than that.

The data is a bit stale though, so I triggerd a new export. I just want to verify the data integrity before I upload the new set. Will be a few days. I'll close the issue when it's done, and maybe make a note in the readme about the state of things.