Guidance for publishing orchard packages in custom repository

[jersiovic]

When you use Orchard you have to decide if you use the source code to be able of fixing things on your own way without having to wait an approval of a PR, like I would wish with this for this issue https://github.com/OrchardCMS/OrchardCore/issues/6228. Problem is I pay every day this advantage on compilation time and resource consumption of visual studio with the big solution loaded.

The other option is to take myget packages resigning to fixing anything quickly to Orchard source code in exchange o good performance when you work on your solution.

So, I want bet of both worlds. Is there any guidance for compiling orchard modules and publising them on a custom packages repository that is the one I will use for my solution?

[hishamco]

Problem is I pay every day this advantage on compilation time and resource consumption of visual studio with the big solution loaded.

I'm suffering fro this too ;) I think you need to work without the source code, for the most cases adding a custom module may enough, according to the issue that you mentioned above this not very straightforward, because the change on the Core APIs, probably subclass the PortableObjectStringLocalizer with the other changes that I did can works fine

[Piedone]

You can publish "releases" for yourself to a custom feed, also from a CI environment. This is what we do at Lombiq too.

[Piedone]

Why is the nightly build not actually published to nuget.org BTW? @sebastienros ?

[jptissot]

I would love to see a monthly build on nuget.org. I wonder if we could do an unofficial build somewhere.

[Piedone]

We can move the nightly from MyGet to Cloudsmith because it offers 100GB storage for open source, enough to have a looong time before they roll out. Or set up a static mirror with Sleet, or just publish the nightly to an Azure Blob directly and ditch MyGet (MyGet has a browser for packages though, however, maybe that's not really needed).

[lskillen]

You'd be very welcome over on @cloudsmith-io - let us know if we can help. :-)

[Piedone]

Checking out the referrals in the site analytics, are we? :D

I just quickly looked at it but I think your service would be super useful here. Basically, the issue is that the packages of the nightly build currently roll out very quickly on MyGet, becoming inaccessible within days. So unless you want to use the outdated version from NuGet (why it's outdated is a separate issue) you either have to run after the updates or use your own package source.

Instead, it seems to me that we could have a much longer history of packages for this dev feed on Cloudsmith. Each build output is around 30MB, so that's >3000 versions. Currently, we have maybe 3 such builds a day on average (see MyGet). Even with this, every such release could be kept for years. Alternatively, we could change it to be an actual "nightly" build, producing just one set of packages each day (if there are new commits).

[lskillen]

Checking out the referrals in the site analytics, are we? :D

😊 You got me! 😁

I just quickly looked at it but I think your service would be super useful here. Basically, the issue is that the packages of the nightly build currently roll out very quickly on MyGet, becoming inaccessible within days. So unless you want to use the outdated version from NuGet (why it's outdated is a separate issue) you either have to run after the updates or use your own package source.

Instead, it seems to me that we could have a much longer history of packages for this dev feed on Cloudsmith. Each build output is around 30MB, so that's >3000 versions. Currently, we have maybe 3 such builds a day on average (see MyGet). Even with this, every such release could be kept for years. Alternatively, we could change it to be an actual "nightly" build, producing just one set of packages each day (if there are new commits).

Yep, I think that makes sense. We've got other open-source users that do something similar, plus the same thing with other package formats that have snapshots (e.g. Java/Maven). If you'd like to keep it tidy on purpose, there's retention controls to trim the fat.

You don't need pre-authorisation to create an open-source repository at Cloudsmith. You can just create an account and specify "open-source" as the repository type. We did it that way to support companies that occasionally have both private and public (or open-source) repositories.

[sebastienros]

This problem has been discussed already and we found some solutions.

We tried to use GH package repository, and it's done on every build (https://github.com/orgs/OrchardCMS/packages). The issues are that to reference the feed we need to be authenticated. Second issue is that there is also a limit in bandwith.

Then we thought about hosting the feed ourself so we are not limited by free offers. We found several projects, including this one which I think is the best solution for us: https://loic-sharma.github.io/BaGet/

Next is to set it up.

[sebastienros]

Now if cloudsmith is really only limiting the project to 100GB and 1TB I don't see why not use it. Until they don't offer that anymore or kick us out ;)

[lskillen]

Now if cloudsmith is really only limiting the project to 100GB and 1TB I don't see why not use it. Until they don't offer that anymore or kick us out ;)

Yep, that's the default. We've offered higher to projects that have needed it, but usually, this is more than enough to get going. It'd be a bit rude of us to kick you out! ;-)

Public feeds in Cloudsmith are properly public. You don't need to be authenticated.

[Piedone]

A hosted solution (until they inevitably kick us out :D) is much better than having to nurse something ourselves.

Cool, so according to Lee this can be set up straight away. It seems to me that I can just set up an Orchard Core org and add people. If anybody has any objections let me know, otherwise, I'll do what needs to be configured there during the weekend, then the CI will need to be changed.. Is there any central Orchard e-mail address or something for such kind of registrations @sebastienros?

[sebastienros]

No special email. Use yours or mine, and add more if possible for backup. You'll need me to update the appveyor file with an encrypted token.

[Piedone]

Cool, thanks.

[Piedone]

We need an organization e-mail. For now, I've used mine but we need some common one. Added @sebastienros, @agriffard, @BenedekFarkas as owners. Let me know if there should be others.

I've set up a package repository and everything so I think the only thing missing is actually publishing the packages. Who has access to the CI for this? Here're the instructions: https://cloudsmith.io/~orchard-core/repos/orchard-core-preview/upload/#formats-nuget (and here are the ones for consuming the feed: https://cloudsmith.io/~orchard-core/repos/orchard-core-preview/setup/#formats-nuget). I've named the repo orchard-core-preview to mimic the existing one but we could potentially have multiple ones. Maybe use the name orchard-core-dev instead so it actually reflects the dev branch?

There is no retention limit now, so all packages ever published will remain forever (until we run out of storage but I only see that you can set a much smaller limit of 20GB; @lskillen can we just say to delete old packages when we run out of space?).

Storage for packages is now in North Carolina because. But supposedly there is some edge caching so it won't be too bad anywhere else.

We can also set up a custom domain for this, like packages.orchardcore.net.

Once we test this setup and it works well I'll also update the docs.

Oh, and BTW the service seems really great, thank you @lskillen!

[jersiovic]

Awesome!

[sebastienros]

Updated in the repos:

nuget sources add -Name orchardcore-preview -Source https://nuget.cloudsmith.io/orchardcore/preview/v3/index.json

[Piedone]

I don't think we're done here yet :). I'll test this out next week and update the docs too. With Antoine we're also working on having a custom domain.

[Piedone]

Does Travis publish now to here or is this syncing from MyGet?

[sebastienros]

I already updated the docs too

It's still publishing to myget too, until most have switched to cloudsmith

[jptissot]

Switched one of our repos, works well :)

[agriffard]

Do they provide badges so we can change it in the ReadMe?

[Piedone]

Great then!

Badges: I'd swear I've seen this somewhere but kind find anything. @lskillen do you have badges that we could add to the Readme like the NuGet one you can see here: https://github.com/OrchardCMS/OrchardCore/blob/dev/README.md?

[lskillen]

Sure thing! See here: https://cloudsmith.io/~orchardcore/repos/preview/packages/detail/nuget/OrchardCore.Application.Cms.Targets/1.0.0-rc1-13293/#badges

The bottom markdown on the page gives you the "latest" version for that package, but should be OK for your purposes of a README badge. It's worth bearing in mind that GitHub caches badges, but there are ways to invalidate (can cover that later).

[Piedone]

Ah OK, I should've looked under a given package, not the repo, thank you!