Organize and track broker versions

[troyraen]

Goals:

• Design a system to assign a global version for the code in this repo. (decided on semantic versioning)
• Collect the broker version, and versions of other important packages, in the metadata tracking system. (put __version__ init file)

To do:

• [ ] Create a dedicated file to hold the broker version, and a method for other files to retrieve the version.
• [ ] Create the process by which the version gets bumped. ~Perhaps this should be handled by a GitHub Action, triggered by a merge into master.~ Consensus is that it's best to bump versions manually... Decide/document the when (merge with master?) and how (major/minor/patch).
• [ ] Configure broker components to collect relevant versions and add them to Pub/Sub messages as metadata attributes. The broker's global version may only need to be collected once (i.e., by a single component). Other components need to collect versions of packages important for tracking provenance (e.g., SuperNNova component needs to collect the supernnova version).
• [ ] Complete #78 so that the pgb-utils version doesn't conflict/cause confusion with the global repo version.
• [ ] Update the docs to describe the new versioning and metadata tracking. Also, explain why the broker design still necessitates multiple modules, each with separate requirements.txt files (see here).

Open question: Should broker-utils be moved into its own repo? Every broker component except the consumer relies on these tools. Since each component is deployed to a separate location in GCP, broker-utils is pip-installable so that we don't have to copy it separately into each component before deployment. This means that it has its own official version, which will not necessarily match the global version for the repo.

This Issue was prompted by the discussion in PR #72.

See AMPEL's "analysis schema"

[troyraen]

Re: the "open question" above.

My initial feeling was that broker-utils is such an integral part of "the broker" that I really wanted to keep it in the same repo and figure out how to resolve the version conflict. I have previously looked at other brokers' repos and seen that they have their user tools in a separate repo, but I hadn't seen anything comparable to broker-utils in a separate repo.

Upon looking more closely:

• Fink has all their science modules in a dedicated repo, separate from the main broker repo.
• ALeRCE doesn't even seem to have one main "broker" repo. There are 49 different repos at github.com/alercebroker and many of them are individual broker components/tools. There is one repo that contains a "framework (...) to create a dockerized pipeline to process an alert stream". So I assume this is how they collect/package/deploy the broker.

Ok, so I'm convinced that just because broker-utils is integral to the rest of the "the broker", it doesn't need to be in the same repo. Moving it probably makes the most sense.

(pinging @wmwv)

[wmwv]

Sounds good. It's good to think through these. I think having 3-5 repos can be reasonable.

There are good and bad features of each choice. One repo can be good, but having to increment the version on everything even when you just change something entirely unrelated is distracting. But splitting things into 100 different repos when really the packages are deeply interdependent is often misleading (Rubin Science Pipelines suffer from this). If you do have a clear separation between parts with a relatively stable API between them splitting into server and client can often be good (like what you have here).

[djperrefort]

Create the process by which the version gets bumped. Perhaps this should be handled by a GitHub Action, triggered by a merge into master.

I've played around with this idea in other projects. I love the idea in concept, but have never gotten it working (with semantic versioning) in practice. Its hard to automate logic to determine if the version bump should be be major, minor, or patch. You can default to one of the three and then manually change it when necessary, but that somewhat defeats the point of automation.

[wmwv]

I agree with @djperrefort . I think declaring versions should remain a manual intentional process.

[troyraen]

splitting into server and client can often be good (like what you have here).

@wmwv What do you mean by "server" and "client" here?

If by "client" you mean the tools developed for external users to access data, I just want to be clear that the broker-utils (aka pgb-broker-utils) package I'm talking about here does not have any user tools. It is strictly for use in the broker code and by developers of the broker code. The pgb-utils package is the user-tools package (its move is separately discussed in #78).

[wmwv]

Right, sorry. I was confused.

[troyraen]

I'm sure my naming of the two packages makes it extra confusing... they are too similar.

[troyraen]

@wmwv @djperrefort Ok, so at this point my feeling is that we should split this repo into 3 separate repos as follows:

1. User tools and tutorials, including the two directories at 1) pgb_utils, and 2) docs/source/access-data

2. broker-utils package, including the directory broker/broker_utils. The only reason I think this should be separate from the main repo is that it's official (PyPI) version will not necessarily be the same as whatever version we assign globally to "the broker". Code-wise, this is an independent unit, and simply pulling this directory out into its own repo should not require any changes elsewhere in the code.

3. Everything else, including "the broker" plus docs and tests.

What do you all think?

(I agree with Michael, it would be nice to have a new name before we split these out... opened #87.)