TSC Subcommittee Meeting (20 Apr 2021): Package hosting solutions

[blag]

April 2021 @StackStorm/tsc 1 hour meeting will take place on Tuesday, 20th Apr 2021, 09:30 AM US Pacific. See #33 for more info about how to join.

Agenda

Other hosting options

Moving to Packagecloud.io OSS plan and complications

• Their current OSS plan
  • 250 GB bandwidth/month
  • 25 GB storage
• In the last 12 months, we have used
  • Bandwidth: 446 GB/month (average), 1.2 TB (maximum)
  • Storage: 182 GB (average), 190 GB (maximum)
• Sweetheart OSS plan restrictions
  • 500 GB bandwidth/month
  • 200 GB storage (need to clean up old versions that we don't support anymore)
  • ❗ If we exceed those limits and the excess "severely impact [Packagecloud's] business finances" they would terminate our OSS plan within 30 days and force us back to an Enterprise License plan ❗
  • This means that we should absolutely have a backup plan in place, or even update our apt/yum repo config files to use something like packages.stackstorm.com, and point that to packagecloud.io in DNS so we can quickly redirect traffic away from them to minimize disruption
  • On the other hand, maybe this is a good way to force users and resellers to support the project monetarily
• Sweetheart OSS plan obligations
  • Add PackageCloud to the StackStorm Partners page
  • Add PackageCloud logo with backlink to PackageCloud as a "Sponsor" on the stackstorm.com website (~500 daily visits)
  • Add PackageCloud badges on GitHub with backlink to PackageCloud
  • Collaborate on a blog post and publish to Open Source community
  • Twitter (3K followers)
  • LinkedIn (1K users)
  • Email Newsletter (13K users)
  • Slack Community (6.5K users) about the PackageCloud partnership
  • Advertising the PackageCloud brand, its services and recognizing the support to our community
    • This is not well defined
  • Quarterly tweet/social media post linking to our quality and popular blog posts

Moving to Packagecloud self-hosted plan and complications

• Free?
• Host on our AWS infrastructure
• Unlimited bandwidth (pull from AWS OSS credits)
• Unlimited storage (pull from AWS OSS credits)
• Manage it ourselves

Moving to Cloudsmith for package hosting

Cloudsmith has reached out regarding becoming a StackStorm partner via hosting StackStorm packages.

• Reasons to use Cloudsmith
  • Actively developed and maintained
  • Reactive to issues/requests
  • Support, as in, from real Humans, is world-class, fast, and frequently involved with users
  • Offers proper org/team role-based access controls, for better setting up secure pipelines
  • Has support for 22+ package formats, to cover any type of deployment or distribution need
  • Has a specific focus on security and securing the supply chain; influences features/roadmap
  • Performance is a top requirement
  • Powered by a worldwide CDN and multi-region infrastructure
  • Actively pursues annual pen-testing, provides a detailed security policy, targetting SOC2/ISO
  • Offers significant support to open-source companies, projects, and other partnerships
  • Some existing partners are already Cloudsmith customers
• Downside
  • Need to update the installation script to point to a different package repo
• Questions
  • Can we host old packages (eg: v3.4.1 and before) on Packagecloud and new packages (eg: v3.5 and later) on Cloudsmith?

Self hosting our own package repos

• Most flexible solution
• Uses AWS credits (shouldn't be a big deal; we have plenty and can get more)
• Operational overhead
• Ready made Docker image to host on AWS S3
• Ready made project to host packages

Host a redirect server

• We should probably implement this anyway, since our package hosts might not give us a lot of time before we either have to start paying, or our traffic gets slowed/cut
• Very little load and traffic (just requests+redirect responses)
• Uses AWS credits (shouldn't be a big deal; we have plenty and can get more)
• DNS redirection?
  • Relies on URL paths to be the same between different package hosts Example: pghost1 hosts at https://pghost1.com/packages/<repo_type>/<os>/<os_version>/<package_name> pghost2 hosts at https://pghost2.com/deb/<os>/<os_version>/<repo_type>/<package_name>
• HTTP redirection?
  • Can redirect to different URL paths
  • Can easily collect analytics for ourselves (nginx+greylog or something)
  • Can ignore traffic from our own CI tests
  • Easily secured (just nginx exposed to the internet with a very simple configuration)

Deliverables

• [ ] A plan to switch (or not) package hosting providers
• [ ] A delegate responsible for overseeing the switchover and/or redirect implementation
  • [ ] Need access to credentials in 1Password?
• [ ] A timeline for when this will be done (we lose $$$ every month this isn't completed!)

[blag]

• Need to get details on what Cloudsmith is willing to host @nmaludy (@blag will connect you)
  • See: https://help.cloudsmith.io/docs/open-source-hosting-policy
  • Are they prepared for 1.2TB/month worst case?
  • How much would this cost?
  • What amount of social media engagement would this require?
• Follow up with Packagecloud about their self-hosted plan @blag
  • Do they offer this on a free/OSS plan?
• Create a Deb/RPM file that just configures the .list/.repo files, imports the GPG key - @nmaludy
  • Makes it easy for us to use
  • Who hosts the repository Deb/RPM?
• HTTP redirecting is probably too much overhead, DNS redirect would work record
• Need to implement DNS redirection to prototype it
  • Add a CNAME record to point to their (Packagecloud or Cloudsmith) A record
  • Need to update our installers to point to our new packages.stackstorm.com subdomain
  • GPG key signing of repo metadata will make this not work

[nmaludy]

Here are the requests from CloudSmith:

As discussed on the call, there are quite a few marketing activities we'd love to get you guys involved in as part of the sponsorship. I've included them below for you guys to discuss in your meeting in early June.

• StackStorm logo usage on our marketing materials, including website
• Cloudsmith logo included within StackStorm Github and website
• Blog post of migration from Package Cloud to Cloudsmith (including promotion on social)
• Quote/case study to include on Cloudsmith marketing materials

Other activities you mentioned that we'd like to also explore include:

• A Cloudsmith integration pack on your site
• Joint events since StackStorm have so much going on there!
• You also mentioned helping with a Puppet integration which would be awesome if you're still interested

[arm4b]

Thanks, @nmaludy for more details. Are there any limits in terms of bandwidth and storage in the FOSS plan Cloudsmith offers us?

[blag]

Current TODO

Note: There is currently nobody signed up to perform these, so please volunteer if you have the time!

• [ ] Completely clear out unneeded packages from the following Packagecloud repos:
  • [ ] staging-unstable
  • [ ] unstable
  • [ ] staging-stable
• [ ] Save older packages (eg: v3.3 and previous) from the stable Packagecloud repo
  • [ ] Put them in an S3 bucket? Could also store then in Amazon Glacier to really reduce costs, since we likely won't ever ever need them again
  • [ ] Remove older packages (eg: v3.3 and previous) from the stable Packagecloud repo

This will give us a better idea of what our useful storage and transfer requirements will be, and whether or not we can fit into a sweetheart OSS plan with Cloudsmith or Packagecloud.

[lskillen]

Thanks, @nmaludy for more details. Are there any limits in terms of bandwidth and storage in the FOSS plan Cloudsmith offers us?

Hey @armab / @blag!

I work at Cloudsmith, so I can help answer this. Based on our discussion before, we're more than happy to work with you on it, because we ❤️ StackStorm. We offer 250GB of bandwidth for unverified OSS users, up to 1TB of bandwidth for verified OSS users, and we can agree on amounts beyond that for sponsored projects such as StackStorm. So basically "as much as you need" with a bit of head-room. Happy to have a discussion on how we can help, but I don't think you'd be disappointed; sponsored projects don't run the risk of getting cut-off, ever. Plus we can help to power a custom domain. :-)