Bundled Packages

[confused-Techie]

Requirements

• Filling out the template is required.

• All new code requires tests to ensure against regressions.

  • However, if your PR contains zero code changes, feel free to select the checkmark below to indicate so.

• [X] Have you ran tests against this code?

• [ ] This PR contains zero code changes.

Description of the Change

WIP: This PR aims to allow better handling of Bundled Packages at the backend level. With the following goals:

• Remove all bundled packages from the DB
• Allow easy managing of Bundled Packages information as it appears on the backend
• Ensure we don't ever have to manage publication and version management of bundled packages
• Ensure these packages never appear as installable
• Avoid wasting resources interacting with the DB when returning information of these packages
• Avoid ever returning update requests for these packages
• Avoid listing bundled packages in search results
• Ensure the names of these packages are always reserved

[Daeraxa]

• Each file must have a top level property readme that contains the README.md of the package.
• Each file must have a top level property metadata that contains the important bits of the package.json.

Is there any way we can handle this more programatically? Honestly I'm not sure what this could look like at a logical level but I'm all for reducing data duplication where at all possible - this could easily get out of sync as and when PRs are made to our package readmes etc.

[confused-Techie]

@Daeraxa Thanks for the review!

But that's a fantastic question.

At the simplistic level, I'd like to create an action that clones the pulsar repo locally, and scans through the packages directory to grab the majority of this information. But there's three big concerns that I don't know how to handle.

• What to do about core non-bundled packages? Those that don't exist in Pulsar yet. We could rely on the repository key being set, but does feel clunky, and would mean the file has to already exist.
• When do we do this update? Could we detect changes to those repos in CI over there and trigger an action here? Do we just set it to happen on the 16th of every month and assume the 15th will stay as release day?
• Do we care if incorrect data is returned to those using an older version of Pulsar? I know they can get the right data still in settings-view but I wonder if that's a concern? Likely not a big one, since already this data is the most up to date, even if not compatible with a specific users install of Pulsar for community packages

Feel free to expand on this or provide ideas, and I'll check in tomorrow

[Daeraxa]

• What to do about core non-bundled packages? Those that don't exist in Pulsar yet. We could rely on the repository key being set, but does feel clunky, and would mean the file has to already exist.

I'm tempted to say that in this hypothetical programmatic approach, for simplicity, these would remain manual - then when they do become bundled the data can overwrite. Alternatively could they be pulled in as submodules or something and do some other kind of magic?

When do we do this update? Could we detect changes to those repos in CI over there and trigger an action here? Do we just set it to happen on the 16th of every month and assume the 15th will stay as release day?

Can we use actions to just check for commits being made to those packages? Specifically to the files we are interested in (readme, package.json)?

Do we care if incorrect data is returned to those using an older version of Pulsar? I know they can get the right data still in settings-view but I wonder if that's a concern? Likely not a big one, since already this data is the most up to date, even if not compatible with a specific users install of Pulsar for community packages

As we don't maintain an LTS type release or support older versions I don't think this is an issue. At most there could be a disparity between rolling and regular but I don't think this should be an issue, the PPR should purely reflect the actual state of itself regardless of what might be in a release.

[confused-Techie]

@Daeraxa You make good points. The only real thing of substance I have to add currently is that I know for a fact that we can optionally trigger CI Actions on the Pulsar repo depending on the path of where changes have occurred. But I do not know if it's possible for us to do that here.

But glad we seem to be on the same page about a solution to versioning, that is there is none. Which seems perfectly fine to me.

---

I'll keep playing with this, since of course this PR needs to actually be utilized elsewhere in the code. But then of course seeing as to how we can keep it up to date with as minimal effort as possible. For a first pass we may even want to wait on that detail to simply have it functional

[confused-Techie]

Well @Daeraxa I've found a way to auto update the Bundled package information, which is present on my last commit. With the only unresolved question of being how we trigger this, but glad to see we have something that seems to be working