search

dizzyd / mcdex

Minecraft Modpack Management
Apache License 2.0
75 stars 8 forks source link

Database Refreshing Out of Date #42

Closed matt-claycomb closed 5 years ago

matt-claycomb commented 5 years ago

I have a Docker image built using this for easy updating of a pack we have on Curse and run a server of. I am running into issues where an update is pushed and approved on Curse (in this case 6 hours ago), but the mcdex database doesn't appear to refresh that frequently (10 hours in the past, at time of writing).

Is there documentation of how the cache file is built so I may run that locally and be able to pull updates in near real-time from Curse?

dizzyd commented 5 years ago

Hi @matt-claycomb -

The crawler is not publicly available as I do not want people inadvertently creating significant load on the CF servers by trying to do real-time polling. This is in part because the CF servers do not update in anything close to real time. From what I've seen things can show up on their site immediately, but from an API perspective it can be several hours before it's visible (weird, I know). The other reason I do not release the crawler is that it's using an undocumented API which has/does change on a regular basis and I don't want to deal with all the support issues therein.

The best possible outcome here would be to get CF to open a real API endpoint that could be polled/monitored in real time, but I doubt they'll do that.

I'm sorry I can't help you beyond this. :/

dizzyd commented 5 years ago

Of note, the crawler does run a few times per day, but there was another API break that caused it to miss files over the past few days. I've corrected this, so it should be more timely moving forward.

matt-claycomb commented 5 years ago

I understand - I wasn't even sure if it was an API or if you actually crawled and scraped data from the human-readable pages.

I just had my concerns as I had seen the db multiple days out of date in the past, and wanted to make sure it wasn't something like a 1 week interval.