search

mauroc / squiddio_pi

squiddio_pi
3 stars 13 forks source link

MIA: deployment strategy #71

Closed leamas closed 4 years ago

leamas commented 4 years ago

Sorry for long post. TL;DR There is a lot of code duplication when it comes to ci builders and deployment.

Currently squiddio deploys into github and cloudsmith. This is as planned while we try to merge the new automatic installer and existing manual package deployment. However, its IMHO time to take a look at the situation.

From a requirements view I can see three separate deployments views:

For now, we have the manual artifacts on github and the installer stuff on cloudsmith. The cloudsmith stuff needs to be divided into stable/unstable, this is under way.

Now, the question is if we should keep deploying into two instances (github and cloudsmith). Obviously, this is more complicated than using just cloudsmith (I don't think using only github is feasible, it's too limited for our three usecases above).

I have no interest in pushing this, just want to raise the the issue. I played with the idea of deploying manual packages at https://cloudsmith.io/~alec-leamas/repos/opencpn-plugins-pkg/packages/ just to see how it could look. Note that it is possible to bundle information with the packages like summary and description, look for previous/next version of a package etc. Also note that the github UI while sort of primitive is really clean.

Part of the problem is that we build the packages both on circleci and travis. In the long run, this is a lot of duplicated code, perhaps too much.

Thoughts?

rgleason commented 4 years ago

Alec, Thank you for looking at the big picture here. There is a lot to consider. I've looked at Cloudsmith for all Deployment including manual deployment as you suggest.

Collaborators affected: Kathi (website), Dave (head programer), PI authors Users: Not affected as the downloads are accessed through the download page. Pros:

Cons:

Dave (Head Programer)

Kathi (Website Download Page)

PI Authors Questions:

Users

Alec I can see the advantage of using cloudsmith as a single deployment and to reduce the size of deployment during transition. I might support it, but there are some outstanding questions that need to be figured out for Plugin Authors. Perhaps you know enough about it to answer?

Perhaps we should try some tests to see how it would work for PI Authors.

rgleason commented 4 years ago

How important is that "commit" link from the "Release" version?

See https://github.com/mauroc/squiddio_pi/releases See the b82d6d9 link ? and the Tag v1.0.17 link taking you back directly back to that commit? Git is a highly integrated system, Circleci and Cloudsmith don't have this capability, they are one way...deploy.

Regarding Description in Github Releases, please look at my fork of Sean's repos, in the Releases Tab celestial_navigation_pi https://github.com/rgleason/celestial_navigation_pi/releases weather_routing_pi https://github.com/rgleason/weather_routing_pi/releases weatherfax_pi https://github.com/rgleason/weatherfax_pi/releases watchdog_pi https://github.com/rgleason/watchdog_pi/releases climatology_pi https://github.com/rgleason/climatology_pi/releases and the list goes on.

Sean and I have a long time relationship. I have helped him build several of these plugins too, particullarly climatology and weatherfax. I submit PR's sometimes he accepts sometimes not, sometimes he is sailing or working on other projects. Eventually he comes back and things get done. Meanwhile it serves no purpose to wait to release a plugiin for Windows that works. More recently with appveyor and travis we have expanded the offering. That is the way we work. Yes it can be messy but we git it done. I do not know if Sean is interested in curating all his plugins, but that would be fine with me if it could be done reasonably timely.

Are you building a structure I/we cannot work within?

rgleason commented 4 years ago

In the test Opencpn Plugins Pkg the files are named

squiddio-windows-mingw-installer 6.1.2

with the 6.1.2 number being assigned by cloudsmith. I am assuming that the names actually used will be as uploaded including version numbers etc

I also assume that the tags are automatically assigned.

leamas commented 4 years ago

EDIT:

How important is that "commit" link from the "Release" version?

Not important at all unless there is an elaborated commit message. A simple link back to github serves IMHO the same purpose.

What access will I have to opencpn-plugins-pkg

None, unless @mauroc gives you the deployment key. Which I would advice against.

As a downstream maintainer it's not really natural that you deploy to the main repos. It might be better to leave @mauroc in control of that and that you setup your own repo. This is how downstream packaging works, be it on github or cloudsmith.

Are you building a structure I/we cannot work within?

I'm not building anything, I'm raising an issue. If you think the current way to handle released, manual (so to speak) packages is the way it should then that could be part of a strategy, for example

Anything is possible, it's basically a functionality/maintenance trade-off. But we need more input here, notably @mauroc.

leamas commented 4 years ago

with the 6.1.2 number being assigned by cloudsmith

Actually, no. It's assigned by the circleci-upload script.

But you are right, the last digit is missing. After all this is just a test shot. I'm starting to put things into https://cloudsmith.io/~alec-leamas/repos/opencpn-plugins-stable, with a correct version.

rgleason commented 4 years ago

Alec, I am strongly leaning towards leaving it as is, if I cannot create releases of Sean's many plugins. This has been a long term condition, it is the way we have been working, and changing it will affect many Windows users over time. I understand that that goes against rules, but it is a "workaround" that gave us most of the main windows plugins (any that did not have active authors + sean's) for OpenCPN v5.0.0, built by me within a short time of the release.

If you can explain a way to modify the cmake and scripts to deploy/ push a release to a my cloudsmith repository, and I can get that working, that would be fine, otherwise my vote is to leave it alone..

rgleason commented 4 years ago

PS I could care less about the amount of repository space used, at this point.

leamas commented 4 years ago

Deploying to your repository is simple, it's basically just to patch REPO= in ci/circleci-upload.sh, soon to be STABLE_REPO= and UNSTABLE_REPO=

Actually, it's also simple to add a patch which makes it possible to use these variables in the environment (if they exists there). This way you could deploy to your own repo without any code patching at all by using some variables set up in cloudsmith.

If you intend to use also make your installer artifacts usable you also need to patch PKG_BASE_URL in CMakeLists.

rgleason commented 4 years ago

OK. I trust you. I'll buy that, but I I'd like to understand how to do it at some point here. I like the idea of Stable_repo and unstable_repo too.

If and when you decide to go ahead with this, doesn't it make sense to save all your work and scripts for upload to github release, somewhere, somehow?

How much more work is this?

I hope Bdbcat, Kathi and some of the Plugin Authors will chime in here. We started using the Github Release Tab because it was available and relatively easy to use. What does Mauro think? What does Jon Gough think? He's been using git Release extensively!

leamas commented 4 years ago

This is not a question of going ahead with anything. It's a question how we want to deploy things, one way or another. In particular, I see a need for separate deployments for users (stable, manual packages), developers (everything) and the installer (stable xml + tarballs).

In the end we need to decide how to handle these three deployments. This is what this issue is about. Scripts and so on: just forget it for now.

leamas commented 4 years ago

doesn't it make sense to save all your work and scripts fo

We are using git. Once it's committed and pushed it's saved.

rgleason commented 4 years ago

Alec I wrote a note to some of those involved, to provide you with some feedback and to make them aware. I hope that's ok. I think/hope you'll have good support for this.

rgleason commented 4 years ago

One thought about having a single deployment for opencpn-plugins on cloudsmith, Will there be so many different plugins that users manually looking and downloading have trouble? Or do we just rely on the Opencpn.org Download page?

leamas commented 4 years ago

Will there be so many different plugins that users manually looking and downloading have trouble?

Definitely. Which means the if we go for using cloudsmith also for "manual" packages, these must live in a separate repo.

This makes three repos: installer (stable xml + tarballs), manual (stable packages for manual installation) and unstable (everything, probably a mess but only for devs).

mauroc commented 4 years ago

apologies for chiming in late. I am glad @leamas raised the issue, as i find the current setup (cloudsmith+github) unnecessarily confusing and, in general, a recipe for trouble. Keeping two separate download sites can only lead to inconsistent versions, outdated packages, code complexity, operator error on the part of the plugin developer, and more importantly a bad UX. My vote is to keep everything in one download site (cloudsmith seems like the obvious choice), and filter out complexity from the end user by using selective pages, as suggested above.

Also, worth noticing that if you google "download squiddio plugin" , you most likely end up on an opencpn.org page, which means the web site editor could further organize and simplify the links. Isn't that the kind of UX we want, where our web site is the primary way to access manual downloads, rather than a third party site? My 2 cents...

mauroc commented 4 years ago

PS - I appreciate that the opencpn.org download page will never be quite up to date as cloudsmith, as that would require too much manual updating. We could recommend, however, that the links on the .org site point - for instance - to the latest minor release, and a generic link at the bottom would point to cloudsmith and say something like "click here for more platforms or to access the latest builds"

rgleason commented 4 years ago

Good points Mauro. When the Plugin Installer is used by everyone, the correct download is automatically downloaded for the given OS and OpenCPN version, so some of these issues will go away.

Question: Are the Cloudsmith files going to be in separate directories for every major Opencpn Version? IE 5.0, 6.0 ...or what? Or does this create trouble for PI authors redirecting...?

leamas commented 4 years ago

My vote is to keep everything in one download site (cloudsmith seems like the obvious choice), and filter out complexity from the end user by using selective pages, as suggested above.

Well, you are the upstream maintainer, so your vote is not just a regular one. IMHO, this would implicate three cloudsmith repos:

Let's sleep on it and go in this direction unless someone comes with good arguments against it within a day or two. Personally I think it's the best option. That said, again, I really have no interest of pushing anything besides that there is some kind of decision.

leamas commented 4 years ago

Are the Cloudsmith files going to be in separate directories for every major Opencpn Version? IE 5.0, 6.0

I don't see any sane reason to organize it this way. cloudsmith has powerful search and sorting capabilities whach can be used iff the plugins are in one repo. Some README info about required ocpn API version should be enough. After all, trying to load incompatible plugins are flagged by opencpn.

rgleason commented 4 years ago

Good.

rgleason commented 4 years ago

Maybe the title should be Plugin Installer backend (Draft)? https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:developer_manual:plugins:plugin_installer

leamas commented 4 years ago

@mauroc: there has been no more input here. Does this mean that we should decide on a simple "deploy everyting using cloudsmith" strategy with three different repos according to https://github.com/mauroc/squiddio_pi/issues/71#issuecomment-545132864 ?

You are the maintainer. The decision is certainly yours to make.

mauroc commented 4 years ago

Let's sleep on it and go in this direction

well, I slept on it, so I thought we were already going in that direction :-). Seriously, I think it makes sense. I suppose the reason you can't use the stable repo for manual is that you don't want all the extra xml files to confuse the user? The executables are going to be identical, right?

I will create a mauro-calvi/manual repo on cloudsmith

rgleason commented 4 years ago

Mauro is making his own decision, but I think I might stick with the github release deployment for the manual deployment because that is done by plugin and would be easier to find the right file for K. Since mauro's cloudsmith repos is just for squiddio, it will be easier to find the right file.My 2 cents.

-------- Original message -------- From: Mauro Calvi notifications@github.com Date: 10/24/2019 17:20 (GMT-05:00) To: mauroc/squiddio_pi squiddio_pi@noreply.github.com Cc: Rick Gleason fcgleason@tgp-architects.com, Comment comment@noreply.github.com Subject: Re: [mauroc/squiddio_pi] MIA: deployment strategy (#71)

Let's sleep on it and go in this direction

well, I slept on it, so I thought we were already going in that direction :-). Seriously, I think it makes sense. I suppose the reason you can't use the stable repo for manual is that you don't want all the extra xml files to confuse the user? The executables are going to be identical, right? I will create a mauro-calvi/manual repo on cloudsmith

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe. [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/mauroc/squiddio_pi/issues/71?email_source=notifications\u0026email_token=AASBEN3I6IOLHXU24F5OU33QQIGR3A5CNFSM4JCTUHUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECGPICA#issuecomment-546108424", "url": "https://github.com/mauroc/squiddio_pi/issues/71?email_source=notifications\u0026email_token=AASBEN3I6IOLHXU24F5OU33QQIGR3A5CNFSM4JCTUHUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECGPICA#issuecomment-546108424", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

mauroc commented 4 years ago

You are the maintainer. The decision is certainly yours to make.

of course I am only making decisions for the squiddio plugin. However, I thought this was the model for the process that all plugin admins will eventually use, each one creating their own circleci, appveyor and cloudsmith accounts. Is that correct?

I think @rgleason is much more qualified to make decisions for the entire opencpn community. Of course, if you guys decide we are not discontinuing github releases, then that is perfectly fine with me.

rgleason commented 4 years ago

IMHO I think it is fair to ask Kathi about this as she and Dave are doing the website... However I just tried searches in cloudsmith and they are reasonably fast. If all plugins could eventually end up in one single repository "opencpn-plugins-manual" I have no doubt that she would like that! You could ask her, I think you have her email or it can be sent via this page https://opencpn.org/OpenCPN/info/send.html Pick "Site Content"

leamas commented 4 years ago

I suppose the reason you can't use the stable repo for manual is that you don't want all the extra xml files to confuse the user?

Yes.

The executables are going to be identical, right?

The .exe, .deb, .img packages etc: yes.

However, I thought this was the model for the process that all plugin admins will eventually use

Perhaps. But in the end it is up to each plugin maintainer, and I guess they might come to different conclusions. After all, as long as users can find their packages and the plugin installer as well you can't really say that one way or another to deploy things is plain wrong. It's just a question about what's suitable.

If things goes in the direction I see (they seldom do...) I guess there will at least be long transition times. In the end, I guess the squiddio community makes it's own choice.

rgleason commented 4 years ago

I think I see your point, I thought you were planning:

If so, maybe those this same URL for "opencpn-plugins-stable should be used by Kathi for the website download page and any users doing manual install? In that case, the only two things I would really like to see are adding a windows "exe" file with "ov50" in the name (not thinking about linux and mac users at all here).

The plugins using old deployment code and manual techniques will just continue as in the past until they convert to the new Plugin Installer.

I predict that the transition will be very fast and users will clamor for the new system if plugins don't support it.

--Does this schema create permissions problems? --Would it make it easier to configure cloudsmith for the "maintainers"? --Would the central repository like https://cloudsmith.io/~alec-leamas/repos/opencpn-plugins-stable reach the Cloudsmith limits for opensource and macos builds?

Ideally there should be a consistent schema, for the benefit of all.

leamas commented 4 years ago

I think I see your point, I thought you were planning:opencpn-plugins-stable [...]

I have no intention whatsoever to centralize plugin repositories. It would be a nightmare to little benefit for users at this point.

TBH, my quest here at squiddio is more or less completed. The installer artifacts are installed in sane locations which together with their xml metadata. This is all required from the installer's perspective.

However, in the process to get here we have ended up in a ci & deployment mess. I feel responsible for this mess, and is thus willing to help cleaning up the situation. This issue is the first part of this. In order to know how to clean up builders and deployment we need to know what to deploy where.

Yes, I have proposed using three cloudsmith repos based on @maurocs initial response in https://github.com/mauroc/squiddio_pi/issues/71#issuecomment-545070616. However, this is not important for me. The only important thing is that the community (in the end, @mauroc) actually makes a decision. When we know where to store things we can clean up builds and deployment, but not before that.

To me, it looks like the possible open issue is where to deploy the manual packages currently on Github . Should they stay there, or should we have them in a separate cloudsmith repo? Can we reach a decision on this I think it would be really helpful.

rgleason commented 4 years ago

Ok, knowing there is no central deployment that is an easy answer I think (knowing that @mauro is making the decision). The current manual deployment situation is somewhat various and not consistent. For awhile Pavel had a system, however there are more plugins and authors now. Please make a schema that includes a consistent repository in Cloudsmith under each plugin maintainer for manual packages.

I now understand that each plugin author/maintainer will have to:

  1. Create their own free Opensourc accounts on GitHub, Travis, Circleci, Cloudsmith.
  2. Configure authentication keys so the system works properly.
  3. Make sure that the system works properly
  4. Use the --tag to deploy stable code.
  5. Use the "manual" repository for plugins installed manually.
  6. Kathi and Dave will have to be advised via email when there is recent activity on the manual repository to be able to update the website Download page.
  7. The website download page will have to be updated with new links to Cloudsmith when the Plugin Installer code is in place.

Good luck Mauro

mauroc commented 4 years ago

knowing that @mauro is making the decision

again, I don't want to make a decision for the entire opencpn community!

I was asked for some input and ideas, which I provided, and can be mostly summed up as "keep it simple".

I don't want to be on record as the guy who decided to follow a decentralized model. That was the model I was given initially ( I was asked to create my own circleci, cloudsmith etc. accounts), and I went for it.

If it makes more sense to have a centralized repo (and I think there is merit to that in terms of overall simplicity) I am all for it! Not my call guys, sorry. I am just a single plugin developer who agreed to have his plugin (squiddio) used as an initial test case.

leamas commented 4 years ago

again, I don't want to make a decision for the entire opencpn community!

Indeed :)

I don't know where the idea of some central deployment for all plugins idea comes from. As I raised this issue it was about squiddio, nothing else. Any attempt to centralize plugin deployment must really be discussed in the overall opencpn project, not in some random plugin.

For now each plugin deploys things in whatever ways they find suitable. So also squiddio.

From a squiddio perspective is seems that there is one open issue: where to deploy the manual plugins currently on github (everybody seems happy with having the installer stuff on cloudsmith).

We really need to come to a decision on this. As I understand it @mauroc propose that we use cloudsmith also for what's currently on github for simple KISS reasons. I basically concur, although I can live happily with any alternative as long as we just decide on this.

leamas commented 4 years ago

I now understand that each plugin author/maintainer will have to:

Create their own free Opensourc accounts on GitHub,

Deploying on github indeed requires a github account. If deploying elsewhere you still need an account on github, gitlab, bitbucket or so to use git in a collaborative way

Travis, Circleci,

This is something to clean up, yes. To do this we first need to sort out the issue discussed here at #82

Cloudsmith

Perhaps. Or whatever we use to deploy.

Configure authentication keys so the system works properly.

Yes. This is notably easier on cloudsmith which doesn't need encryption

Make sure that the system works properly

(!)

Use the --tag to deploy stable code.

(!!) Using --tags is absolutely standard git. You must always push your tags

Use the "manual" repository for plugins installed manually.

Perhaps, depending on what we decide. This could be an automatic push from e. g. appveyor-upload.sh or so.

Kathi and Dave will have to be advised via email when there is recent activity on the manual repository to be able to update the website Download page. The website download page will have to be updated with new links to Cloudsmith when the Plugin Installer code is in place.

Possibly, unless we maintain a generic "latest release" link.

rgleason commented 4 years ago

That's a good idea!

rgleason commented 4 years ago

Leamas wrote:

It's a question how we want to deploy things, one way or another. In particular, I see a need for separate deployments for users (stable, manual packages), developers (everything) and the installer (stable xml + tarballs).

Later Leamas wrote:

Well, you are the upstream maintainer, so your vote is not just a regular one. IMHO, this would implicate three cloudsmith repos:

One for tagged releases of manual packages, what is on github today.
One for tagged releases of installer packages, also kept for a "long" time.
One for untagged test releases, a mess used by devs only and with a
aggressive retention policy.

What about having just two repositories?

The website download page would link to specific files in the first repository for manual installation.

leamas commented 4 years ago

What about having just two repositories?

It's no big deal, but it might be an advantage for users trying to browse manual packages. I prefer to split it, but could live with any alternative.

rgleason commented 4 years ago

No big deal to me either, but I still don't know how you force a tagged deployment to each of the proposed 3 repositories.

rgleason commented 4 years ago

...GitHub Actions and GitHub Package Registry beta programs. .. GitHub Actions and GitHub Package Registry will be moving to general availability on November 13. ...these services will still be free for open source and include free quantities, and that overages will become paid services on November 13. You can continue to use these services with no changes to your existing workflows. If you’d like to use Actions and Package Registry beyond the included limits, you will need to enable overages,

view Actions pricing here, (Linux, MacOS & Windows) Free all public repositories

and Package Registry pricing here, including the free quantities provided in each plan. Free 500gb, unlimited transfer out in actions, 1gb transfer out outside of actions

Just thought you should know about this option.

"With GitHub Actions, we are able to code, build, test and publish ImageMagick for Linux, macOS, and Windows in a single location without configuring multiple external dependencies. This is a game changer!” Dirk Lemstra ImageMagick maintainer

leamas commented 4 years ago

OK, we now have what looks like an alternative to the travis-circleci-appveyor setup. It's sort of interesting, especially the possiblity to bujild also windows on the same host.

However, it is not related to the issue here. The issue here is how we want to deploy things, be it from circleci, travis or appveyour. Or, later, from Github Actions.

This issue is now blocking almost everything. We don't need more spinning wheels here, we need to agree on a deployment strategy. When we have nailed that down we could take next step: a builder strategy. Which might or might not be to just use Github Actions. But please, don't bring new, irrelevant things into this bug where we really just need to agree on how to deploy things.

leamas commented 4 years ago

I have made a proposal based on @mauroc in https://github.com/mauroc/squiddio_pi/issues/71#issuecomment-545132864.

As I see it, here are two open issues:

If we could limit the discussion to these things we could (perhaps) come to a conclusion, close this issue and move on...

leamas commented 4 years ago

In an attempt to make the discussion a little more stringent: my proposal (and, I dare say, @mauroc's ) is that we move everything to cloudsmith, dropping github as deployment platform.

OK?!

leamas commented 4 years ago

Likewise, I propose that we have different cloudsmith dirs for tagged manual packages and installer stuff.

OK?

leamas commented 4 years ago

The main reason for separate dirs is to keep it simpler for users manually browsing the directory. If we don't split this, each release will be 12 files, most of them irrelevant for users looking for a manual package. In a manual packages directory, a release is just 4 files. It's just way simpler.

The deployment code is separate in any case, so this does not add much to code complexity.

There are also retention policy aspects: the need to maintain old releases could differ between the plugins stuff and manual packages. Having separate dirs creates some wiggle room for this.

rgleason commented 4 years ago

Leamas. I agree. It seems clear to me that the choice should be to use cloudsmith for all three repositoties. I hope that helps.

If github actions can be used at some point in this process, it might really simplify things for PI imaintainers configuring Plugin Installer, with just one service (building macos linux and windows) and fewer authentication issues. I have no idea how much work that would be.

rgleason commented 4 years ago

@Mauro Could you chime in on your preference for squiddio_pi so Alec can move forward? (We have not heard anything from Dave, so I think he's ok/comfortable with whatever we decide.) Thanks.

leamas commented 4 years ago

no big deal to me either, but I still don't know how you force a tagged deployment to each of the proposed 3 repositories

It"s actually simple, just a new case similar to the two others in the upload scripts

leamas commented 4 years ago

OK, we have an agreement. Everything on cloudsmith in three repositories as of https://github.com/mauroc/squiddio_pi/issues/71#issuecomment-545132864

Closing. Please reopen if required.