search

foojayio / discoapi

The foojay discovery api (discoapi) is made to discover java packages (jre/jdk) from different distributions.
GNU General Public License v2.0
111 stars 13 forks source link

Frequent 400 errors #46

Closed mikehearn closed 2 years ago

mikehearn commented 2 years ago

I get HTTP 400s almost every time I do a query. I have to hit the server in a loop for it to work, which eventually it will. It appears there's some non-transactional behaviour going on, e.g. the ephemeral ID is often not recognized immediately after it's been vended.

HanSolo commented 2 years ago

Yep I'm currently working on that...seems production is down...

HanSolo commented 2 years ago

Should be better now, can you please try again?

mikehearn commented 2 years ago

It's not better. I still get 400s multiple times when trying to resolve an ephemeral ID before it works.

I'd suggest rethinking this API design. I don't fully understand the need for the ephemeral IDs - if you want to know how many people are downloading/picking a particular JDK you could just provide the download URL directly and then simply ask API users to ping a different endpoint async that does nothing but write to a log. This would be faster for end users, you'd get the data, and it'd be drastically more reliable as it'd remove the need for global synchronous replication (I assume that's what's going wrong here from the apparent multi-second delay before the ephemeral ID becomes usable). We'd happily do it if it's really helpful for you.

At the moment I'm trying to incorporate this API into a product I'm developing for the JVM community and it's just not reliable enough right now. If this problem isn't resolved soon, my plan is to modify the code in this repository to generate static datasets that the app downloads and searches locally. A cloud API is overkill for this use case, there aren't enough records to justify the availability hit.

mikehearn commented 2 years ago

BTW I'm also seeing HTTP request timeouts.

HanSolo commented 2 years ago

Yep there is definitely something wrong here, I also see this behavior. There have currently be some changes in the infrastructure which seem to break things down. We use the ephemeral id to avoid caching the download links and get at least one possibility to get some statistics. For the time we have that trouble with our production version I could offer you to run against my private server that I use for testing, if you are interested, just drop me a mail at han dot solo dot gg at gmail dot com

mikehearn commented 2 years ago

Thanks for the offer, but this is a product I plan to start distributing in a few months to the client side, so I'm looking for a long term solution. I'll probably do a set of canned queries and ship the results statically.

HanSolo commented 2 years ago

The infrastructure problems should be solved now.