search

spring-attic / bomr

Command-line tool for creating and updating a Maven bom
32 stars 5 forks source link

artifacts and artifacts-delta may trip Central's automated blacklisting #14

Closed wilkinsona closed 5 years ago

wilkinsona commented 5 years ago

I think both artifacts and arifacts-delta may trip Central's automated IP blacklisting resulting in 403 responses. The page that describes Central's behaviour has the following to say:

Most common build tools such as Maven should be requesting specific files and NOT directory listings (which is a common indicator of scraping activity). Make sure that you are not inadvertently making numerous directory requests within a short timeframe to avoid potential blacklisting.

Bomr currently uses the directory listings to look for the artifacts that are available with a particular version in a particular group. We'll need to find another way, remove the commands, or throttle the requests to avoid putting too much strain on Central.

wilkinsona commented 5 years ago

We should use search.maven.org instead. For example, to list all of the artifacts in org.infinispan with a version of 9.4.15.Final, we can use https://search.maven.org/solrsearch/select?q=g:org.infinispan+AND+v:9.4.15.Final. The rows and start query parameters can be used to control the page size and starting point respectively.