search

jpype-project / jpype

JPype is cross language bridge to allow Python programs full access to Java class libraries.
http://www.jpype.org
Other
1.07k stars 177 forks source link

State of the project #760

Closed Thrameos closed 4 years ago

Thrameos commented 4 years ago

We are closing out the work for the targeted JPype 0.8 release. Still a lot of lose ends in terms of bugs, coverage, and hardening, but I believe we are at the point where a feature freeze is warranted.

There are some questions for the release: 1) What should this be called? One user wants us to adopt this as an actual major number making this 1.0. 2) How do we get enough users to test this to make a solid and stable release point? 3) What is not here that is required for a release? 4) Does the dbapi2 interface go in this release or wait for more testing and user input?

This project is actually approaching a production level code so making it a major release with 1.0 makes sense. I have some reservations as we still have some regrettable public interfaces that have yet to be removed. I would like the 1.0 to be a clean point where everything in the API has been reviewed and deemed necessary (or broken and thus removed). We are not yet at that point as there are many external symbols that are legacy which may change from nuisance (ignorable) to in the way (must remove) once I work on the reverse bridge.

Unless something is flat out broken we often don't get reports of problems with a release until months after. But this is supposed to be a good point to last us for at least a year while I work under the hood to provide some new capabilities. We need people to help out to make really kick the tires on this one so that we don't end up with 4 months of fixing issue and patch releases as I am gutting the places needed for the reverse bridge. Github doesn't have a spam the users of a project (likely a good thing), but it does leave us in this type of situation. If we do have a poor quality release and I have already started the next development cycle, then the whole repo is going to end up forked with a release branch and devel branch. We tried that with 0.7 but I could only hold it for months before I had to start pushing thing from devel back to release so that I wasn't constantly needing to task switch (damn didn't I fix this already, wait this assumption is different, no not this one again!)

I have laid out the road map of missing features that we are going for likely requiring 2 major versions of upgrades.

It is still a pretty long road but the end is visible on the horizon and I have a good idea how to finish all of these tasks. It is simply a matter of coding time.

If anyone has a strong opinion on these topics please weigh in. Otherwise we autopilot to the next point.

Thrameos commented 4 years ago

@baztian Do you want to weigh in the version number?

baztian commented 4 years ago

I'm not as involved as I was a couple of years ago. So this is just my two cents.

From both a user and a maintainer perspective I don't like long lived feature branches or some other sort of complex branching models. I like to have fast adoption of features and bug fixes in the master branch and very frequent releases. This goes well together with the semantic versioning approach.

The downside of this is that you either have to keep your api backwards compatible for a longer time or you release major versions frequently and annoy some users as they have to migrate more often.

But I also think that you circumvent some of the challenges you outlined. you'll get more frequent feedback from the user base and it is easier to trace back where an issue has been introduced.

If this project would be driven by me I would release a 1.0.0 as soon as possible. Then I would try to keep backwards compatibility as long as it's no real pain. Always depreciating things so the user base can prepare for upcoming changes easily.

Every new feature should ideally become a new minor version. Every bug fix a new patch level.

BTW: Back in the days I introduced bumpversion (now superseded by the bump2version package) in JPype. There should still be a .bumpversion file in the project... This makes releasing a snap.

Thrameos commented 4 years ago

Thanks very much for adding in bumpversion and for giving your input. After you left unfortunately bumpversion broke, and I just recently resurrected it as we were having many version issues such as unable to track files or missing steps in the release process which having a release procedure including bumpversion helped soleve. All is up and running now so we should be good to go starting with next release. (fingers crossed)

As for the releases cycle I am fine with frequent releases, but all the remaining tasks are each require about 3 months of internal rework apiece. They are also interdependent in nasty ways. If I already have reverse bridge then implementing a Python class in Java is easy as I can pass Python types to Java directly and just need an annotation processor to build the stub interface for JClass to read and build the customized class (as opposed to the current unscalable solution of stuffing the JPype module with all the customizers we come up with.). If it is missing the reverse bridge then I will end up with a very different implementation and likely two apis or have to tear apart the old to make a new. This is sort of why I have 135k lines added and 95k lines removed on 50k line project as building something from scratch can be less work than slowly reorging. Also this type of tying means there is not intermediate state of the code. The major features are likely to break interfaces, as large features require changing implementations which although we don't explicitly change the API (all tested calls are the same), we often have implementation details that change (was returning a list, now returning a tuple, we documented that it would return a sequence).

Thrameos commented 4 years ago

@marscher and @originell Do you have any input on the release?

khaeru commented 4 years ago

I echo the call for frequent major releases.

In iiasa/ixmp#279 and the linked issues, we found that:

Suppose instead the sequence was:

This would have had the effect of signaling to us (the "semantic" in semantic versioning) that we should be alert for possible API breakage from 1.0 to 2.0, but not from 2.0 to 2.1. Instead, we had to decipher this in a more indirect and thus time-consuming way.

I do understand and sympathize with the feeling that the jump from "0.x" to "1.0" is somehow significant, but—as a maintainer of several packages—I've found that when I discard that notion and apply the semver rules literally, things get much simpler and easier.

P.S. I recognize that the notion of what is "compatible" vs. "incompatible" in terms of API changes is fuzzy; changes that might break one project (like ours) might not affect another. So it could help to write in the docs a clear list of what kind of changes will trigger major increments, and what other kinds might be rolled out with only a minor increment.

P.P.S. In case it's not clear, major feature additions can happen in a minor release, as long as there are no changes to existing API. For instance, 1.0 could be followed by 1.1 with the planned DBAPI, and 1.2 and so on; 2.0 would only be necessary if something already in 1.0 is changed.

Thrameos commented 4 years ago

Well starting with the next version we actually have test coverage that is sufficient for detecting API breakage so we could go to semantic versioning. Up to this point I have just been guessing what has changed and nothing on the public API was supposed to have been removed at all for the entire 0.7 series. (We will just call it a miss on that front) It was also unfortunate that 0.7.4 had a source distribution that did not match the actual release. 0.7.4 and 0.7.5 are supposed to be identical.

So I will tally you as a vote for next version being 1.0.0 even through it still contains a number of deprecated methods on the exposed API.

Thrameos commented 4 years ago

@marscher The wheels of bureaucracy finally decided that they would like to review JPype and get credit for producing free software. Unfortunately that means I have to put the next release through my review process to get a release number which will take 2 weeks plus the time to get all the notice statements and copyrights in order and be prepared to verify that no code was included which was not permitted by licensing (ie someone copying code that was proprietary). I have already cleared the profanity checks and export control the last time, but that has expired so I will need to resubmit. It would be most helpful if I have the release code in a final state for the review. Have you decided if this should be the 0.8 or the 1.0 release?

marscher commented 4 years ago

I'd like to pass this decision to you. In my opinion we should stick to semantic versioning asap, as most dependent projects are expecting this. Good luck with the bureaucracy! -- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

Thrameos commented 4 years ago

I am still waiting on the document review forms. Hopefully this week.

ShaheedHaque commented 4 years ago

If I wanted to take the putative 1.0.0 (or 0.8.0..) out for a test, how should I do that? I don't see an obviously named branch. Is it master I should use?

Thrameos commented 4 years ago

The review branch is intended as the proposed release. The internal review of the code was completed this week and the release should proceed once I have made all the required changes.

ShaheedHaque commented 4 years ago

I'm probably missing something obvious...but I don't see a branch called "review" or similar?

Thrameos commented 4 years ago

You will need to use the checkout from PR

https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally

And take PR https://github.com/jpype-project/jpype/pull/776

Example:

$ git clone git@github.com:jpype-project/jpype.git jpype
Cloning into 'jpype'...
remote: Enumerating objects: 12887, done.
remote: Total 12887 (delta 0), reused 0 (delta 0), pack-reused 12887
Receiving objects: 100% (12887/12887), 3.60 MiB | 4.45 MiB/s, done.
Resolving deltas: 100% (9508/9508), done.
Checking out files: 100% (420/420), done.

$ cd jpype

$ git fetch origin pull/776/head:review
remote: Enumerating objects: 316, done.
remote: Counting objects: 100% (316/316), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 409 (delta 307), reused 314 (delta 307), pack-reused 93
Receiving objects: 100% (409/409), 86.64 KiB | 3.77 MiB/s, done.
Resolving deltas: 100% (330/330), completed with 252 local objects.
From github.com:jpype-project/jpype
 * [new ref]         refs/pull/776/head -> review

$ git checkout review
Checking out files: 100% (326/326), done.
Switched to branch 'review'