search

mthmulders / mcs

Search the Maven Central Repository from your command line!
https://maarten.mulders.it/projects/mcs/
MIT License
176 stars 23 forks source link

"MCS ran into an error: failure: failure: failure: failure: failure: failure: failure:" while searching for a class with 2500 results limit. #394

Open ykartsev opened 2 months ago

ykartsev commented 2 months ago

First of all, thank you for a great tool. For some reason I did not find any other way to search Maven repo by full class name (various websites don't work), so it's very useful!

I just built the latest snapshot of MCS (0.7.1) and tried to run the following search:

java -jar mcs-0.7.1-SNAPSHOT.jar class-search -f javax.xml.ws.handler.Handler -l 2500

and got the following error:

MCS ran into an error: failure: failure: failure: failure: failure: failure: failure: 
Error parsing the search result. This may be a temporary failure from search.maven.org.
If the problem persists, please open a conversation at

    https://github.com/mthmulders/mcs/discussions

Make sure to at least provide your invocation of mcs and the version of mcs you're using.

If the error persist, please consider reporting the issue at https://github.com/mthmulders/mcs/issues/new

I must admit that running it with default number of results worked fine:

java -jar mcs-0.7.1-SNAPSHOT.jar class-search -f javax.xml.ws.handler.Handler

resulted in:

Searching for artifacts containing javax.xml.ws.handler.Handler...
Found 2421 results (showing 20)

  Coordinates                                                                   Last updated
  ===========                                                                   ============
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.9.0   02 Jun 2017 at 03:52 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.9.0   02 Jun 2017 at 03:52 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.8.0   20 Mar 2017 at 13:58 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.8.0   20 Mar 2017 at 13:58 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.7.0   14 Apr 2016 at 10:56 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.7.0   14 Apr 2016 at 10:55 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.6.0   22 Jan 2016 at 08:02 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.6.0   22 Jan 2016 at 08:01 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.5.0   07 Apr 2015 at 09:05 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.5.0   07 Apr 2015 at 09:05 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.4.0   02 Jan 2014 at 04:00 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.4.0   02 Jan 2014 at 04:00 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.3.0   14 Nov 2013 at 00:51 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.3.0   14 Nov 2013 at 00:51 (EST)
  org.ow2.bundles:ow2-bundles-externals-jaxws-2.1:1.0.36                        10 Sep 2013 at 04:55 (EDT)
  org.ow2.bundles:ow2-bundles-externals-jaxws-2.1:1.0.35                        19 Dec 2012 at 08:54 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.2.0   28 Nov 2012 at 02:07 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.2.0   28 Nov 2012 at 02:07 (EST)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.1:2.1.0   02 Oct 2012 at 11:50 (EDT)
  org.apache.servicemix.specs:org.apache.servicemix.specs.jaxws-api-2.0:2.1.0   02 Oct 2012 at 11:49 (EDT)

Thank you.

mthmulders commented 2 months ago

First diagnosis: at some point, the search API offered by Maven Central returns an HTML page with a generic HTTP 504 (Gateway timeout) message, rather than a JSON response:

<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>

I think MCS could improve in two ways:

  1. Suggest this issue ("Error parsing the search result") may be caused because the user requests a lot of items. I've added some debug logging locally and observed it happens roughly around ~1000 items or more.
  2. The intro text ("MCS ran into an error: failure: failure: failure: failure: failure: failure: failure: ") should contain fewer failures ;-)
ykartsev commented 2 months ago

Thank you @mthmulders ! And thanks for adding those enhancement issues - really nice to see it's an active project and you take care of it.

I just thought about something a bit related: would it make sense to have an option to only include the latest version of found dependency? Does Maven API support that? That way number of results can be drastically reduced and usability would increase a lot. For example, a class might exist in multiple versions of a dependency, starting from 1.0, hence the search would return all those hundreds of results per single dependency. But if there was a flag to tell "I care only about the latest" - that would be always one per dependency and the list would be much shorter. Worst case, if Maven doesn't support that - results can be requested in chunks (known to not cause this issue for Maven), then arranged by name and version and then latest versions returned. Considering Maven naming convention, this should be doable. I might even try to contribute if Maven does not support that :)

mthmulders commented 2 months ago
  1. Suggest this issue ("Error parsing the search result") may be caused because the user requests a lot of items. I've added some debug logging locally and observed it happens roughly around ~1000 items or more.

To elaborate a bit more: when you specify a limit above 200 (constant defined here, in accordance with the remote API never returning more than 200 items per request), MCS will iterate until it has retrieved all items that match the query or retrieved the number of items you have requested - whichever comes first.

The aforementioned HTML response consistently appears when MCS asks for items 1200 to 1400 in your query.

mthmulders commented 2 months ago

Thank you @mthmulders ! And thanks for adding those enhancement issues - really nice to see it's an active project and you take care of it.

You're welcome, and thanks for the nice words!

I just thought about something a bit related: would it make sense to have an option to only include the latest version of found dependency? Does Maven API support that? That way number of results can be drastically reduced and usability would increase a lot.

That's an interesting feature to consider! Often, you don't care for the gazillion of older versions that are around :-). Would you be so kind to open a new issue for this idea?

ykartsev commented 2 months ago

Would you be so kind to open a new issue for this idea? Thank you, added #403!