Considering supporting checking on a subset of available versions

lread commented 1 year ago

Scenario

Consider a project using Jetty. Jetty supports different release streams based on the major version number.

Version	Years	Home	Min JVM	Servlet	JSP	Status	Protocols
11	2020-	Eclipse	11 (2)	5.0	3.0	Stable	HTTP/1.1 (RFC 7230), HTTP/2 (RFC 7540), WebSocket (RFC 6455, JSR 356), FastCGI, JakartaEE Namespace(1)
10	2020-	Eclipse	11 (2)	4.0	2.3	Stable	HTTP/1.1 (RFC 7230), HTTP/2 (RFC 7540), WebSocket (RFC 6455, JSR 356), FastCGI
9.4	2016-	Eclipse	1.8	3.1	2.3	Stable (Notice)	HTTP/1.1 (RFC 7230), HTTP/2 (RFC 7540), WebSocket (RFC 6455, JSR 356), FastCGI

Upgrading the major version of Jetty can mean significant work, so folks typically stick with one stream until they are ready to upgrade, if ever.

Problem

If I am using Jetty 9.x I am interested in knowing if there are new releases in the 9.x stream. I have no interest in the latest 11.x release, but currently antq will inform me of the latest 11.x release.

Is this just for Jetty?

Well, Jetty is the lib for which I witnessed this problem. Lucene might be another good example. But any lib which expresses breaking changes via its version number would be a candidate.

Idea

I see that antq already supports metadata to describe exclusions. And a command line switch.

Would it make sense to introduce similar to specify that we want to only include certain versions of lib when comparing?

Would a regex for a :antq/inclusions metadata make sense? Not sure how easy that would be to represent in edn, but it is my first idea.

Another idea would be to extend the existing :antq/exclusions so that I could perhaps exclude both the 10.x and 11.x streams.

Next Steps

If you find this idea interesting, I am happy to help in any way that makes sense to you. Lemme know.

liquidz commented 1 year ago

@lread Thanks for your suggestion!

If I am using Jetty 9.x I am interested in knowing if there are new releases in the 9.x stream.

Indeed. It will be useful to support this situation.

In the --exclude option, you can currently exclude specific versions like "foo@1.2.3". It seems good to me to extend this feature to allow wildcards and also enable the same specifications in the :antq/exclude metadata.

e.g.

--exclude=foo/bar@11.x --exclude=foo/bar@10.x
^{:antq/exclude ["11.x" "10.x"]}

What do you think?

lread commented 1 year ago

@liquidz ya I think that would work.

So a trailing .x would be equivalent to matching a trailing #"\..*".

It is certainly simpler than a regex! But I don't think anybody would need the complexity of a regex. And if they do that could be supported somehow in the future.

liquidz commented 1 year ago

@lread Thanks for your confirmation!

But I don't think anybody would need the complexity of a regex. And if they do that could be supported somehow in the future.

If more complex expressions are needed in the future, it would be sufficient to add notations like the one supported in package.json :)

https://docs.npmjs.com/cli/v6/configuring-npm/package-json#dependencies

lread commented 1 year ago

Oh ya, nice, that's good prior work to lean on - if you ever need to!

liquidz commented 1 year ago

NOTE: In the context of semantic versioning, notations supported in package.json seem to be sufficient. But this may be insufficient in the context like git tags that can contain strings other than version numbers.

So, it might be worth to support arbitrary regular expressions.

e.g. ^{:antq/exclude "=~1[01].*"}

lread commented 1 year ago

Ya, regex was in my original thoughts.

Pros:

can express pretty much anything

Cons:

harder for users to express simple things
needs more docs
an incorrect regex could exclude updates a user is interested in

But... I expect folks won't be using regexes to exclude very often. If they need this feature they can also figure out the necessary syntax.

Could antq report on what candidates it is excluding due to user config? Would that help somewhat with regex debugging?

liquidz commented 1 year ago

@lread

Could antq report on what candidates it is excluding due to user config?

Currently no.

FYI I'll cancel the support for regular expressions, as it seems that the original problem can be solved using another way.

liquidz commented 1 year ago

@lread I implemented this feature in feature/exclude-range branch.

Documents are also updated. https://github.com/liquidz/antq/blob/feature/exclude-range/doc/exclusions.adoc#specify-a-version-or-version-range-via-metadata

Could you try feature/exclude-range branch?

lread commented 1 year ago

Sure thing! I'll try it out on cljdoc.

I updated my antq dep to point to your branch - and also mark it as excluded, we don't need the extra noise.

com.github.liquidz/antq ^:antq/exclude {:git/sha "e23d499a4247c1ab91def9ba4bf8d4a1d8689a30"}

Cljdoc currently has the following jetty deps (to override CVEs in pedestal):

        org.eclipse.jetty/jetty-servlet {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.http2/http2-server {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-api {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-servlet {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-server {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty/jetty-alpn-server {:mvn/version "9.4.51.v20230217"}

For a baseline let's run antq without changes, I expect to continue to see the latest jetty release:

|    :file |                                         :name |         :current | :latest |
|----------+-----------------------------------------------+------------------+---------|
| deps.edn |          org.eclipse.jetty.http2/http2-server | 9.4.51.v20230217 | 11.0.15 |
|          | org.eclipse.jetty.websocket/websocket-servlet | 9.4.51.v20230217 | 11.0.15 |
|          |           org.eclipse.jetty/jetty-alpn-server | 9.4.51.v20230217 | 11.0.15 |
|          |               org.eclipse.jetty/jetty-servlet | 9.4.51.v20230217 | 11.0.15 |

Perfect! (the websocket-api and websocket-server libs seem to have been renamed in 10.x so that's why we don't see them reported).

Ok so now let's pretend I am uninterested in 11.x via a change to my deps.edn like so:

        org.eclipse.jetty/jetty-servlet ^{:antq/exclude "11.x"} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.http2/http2-server ^{:antq/exclude "11.x"} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-api ^{:antq/exclude "11.x"} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-servlet ^{:antq/exclude "11.x"} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-server ^{:antq/exclude "11.x"} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty/jetty-alpn-server ^{:antq/exclude "11.x"} {:mvn/version "9.4.51.v20230217"}

I would expect to see the latest 10.x reported:

|    :file |                                         :name |         :current | :latest |
|----------+-----------------------------------------------+------------------+---------|
| deps.edn |          org.eclipse.jetty.http2/http2-server | 9.4.51.v20230217 | 10.0.15 |
|          | org.eclipse.jetty.websocket/websocket-servlet | 9.4.51.v20230217 | 10.0.15 |
|          |           org.eclipse.jetty/jetty-alpn-server | 9.4.51.v20230217 | 10.0.15 |
|          |               org.eclipse.jetty/jetty-servlet | 9.4.51.v20230217 | 10.0.15 |

Awesome!

Ok, but I'm not interested in 10.x either. So I'll update my deps.edn again:

        org.eclipse.jetty/jetty-servlet ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.http2/http2-server ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-api ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-servlet ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty.websocket/websocket-server ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.51.v20230217"}
        org.eclipse.jetty/jetty-alpn-server ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.51.v20230217"}

And now I would expect to see no jetty updates reported:

All dependencies are up-to-date.

Great!

Finally, I'll contrive checking for 9.x updates by downgrading jetty to 9.4.14.x in my deps.edn:

        org.eclipse.jetty/jetty-servlet ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.14.v20181114"}
        org.eclipse.jetty.http2/http2-server ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.14.v20181114"}
        org.eclipse.jetty.websocket/websocket-api ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.14.v20181114"}
        org.eclipse.jetty.websocket/websocket-servlet ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.14.v20181114"}
        org.eclipse.jetty.websocket/websocket-server ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.14.v20181114"}
        org.eclipse.jetty/jetty-alpn-server ^{:antq/exclude ["10.x" "11.x"]} {:mvn/version "9.4.14.v20181114"}

I would expect to see 9.4.51 reported as an available update for all jetty deps:

|    :file |                                         :name |         :current |          :latest |
|----------+-----------------------------------------------+------------------+------------------|
| deps.edn |          org.eclipse.jetty.http2/http2-server | 9.4.14.v20181114 | 9.4.51.v20230217 |
|          |     org.eclipse.jetty.websocket/websocket-api | 9.4.14.v20181114 | 9.4.51.v20230217 |
|          |  org.eclipse.jetty.websocket/websocket-server | 9.4.14.v20181114 | 9.4.51.v20230217 |
|          | org.eclipse.jetty.websocket/websocket-servlet | 9.4.14.v20181114 | 9.4.51.v20230217 |
|          |           org.eclipse.jetty/jetty-alpn-server | 9.4.14.v20181114 | 9.4.51.v20230217 |
|          |               org.eclipse.jetty/jetty-servlet | 9.4.14.v20181114 | 9.4.51.v20230217 |

Perfecto!

My testing above was limited but your change seems to work very well for my use case.

liquidz commented 1 year ago

@lread Thanks for your confirmation! Just released v2.4.1062 :)

lread commented 1 year ago

Awesome @liquidz! Thanks! I will certainly use this new feature!

liquidz / antq