Travis-ci.org shutdown & migration to travis-ci.com

[mliszcz]

I just found out and don't know if this has been discussed already. https://mailchi.mp/3d439eeb1098/travis-ciorg-is-moving-to-travis-cicom

Seems like we'd need to migrate the project to travis-ci.com before end of this year. Probably just some clicking in github and travis project settings is needed.

https://docs.travis-ci.com/user/migrate/open-source-repository-migration#q-what-will-happen-to-travis-ciorg-after-december-31st-2020

A. Travis-ci.org will be switched to a read-only platform, allowing you to see your jobs build history from all repositories previously connected to travis-ci.org.

EDIT 30-11-2020: Below statement is not true. See https://github.com/tango-controls/cppTango/issues/812#issuecomment-735720702. Also, while we are adding more jobs, travis is reducing number of concurrent jobs. Maybe this is an opportunity to look for alternative. https://docs.travis-ci.com/user/migrate/open-source-repository-migration#q-will-there-be-lower-concurrency-for-free-accounts-on-travis-ciorg

[...] free and open-source .org accounts will have concurrency reduced from 5 to 4 concurrent jobs [...]

[jkotan]

Hi Michał,

according to https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing

posted by @cpascual at https://github.com/taurus-org/taurus/issues/1139

the free travis-ci.com service is going to be limited.

As @cpascual suggested to try the native github workflow actions. I've switched my tango tests (e.g. https://github.com/nexdatas/writer/actions) as well as my GUI tests (e.g.https://github.com/lavue-org/lavue/actions) and it works OK.

[mliszcz]

Hi Jan, Thanks for the info! Yes, github actions is something we considered using (we have one new job there already), but nobody has decided yet that any effort shall be spent on migrating all other jobs to github actions or some other service.

I opened this issue so that we at least remember to do the migration and are not surprised that CI is not working once we get back to the work in new year.

[t-b]

I think github actions is the way to go as well.

[t-b]

The way I read it is that now everyone is a potentially paying customer where you just get some free credits to start with. We should definitly move to github actions.

[cpascual]

Hi, I would like to open the discussion about considering moving to gitlab-ci (GL) instead of github (GH) actions.

GH actions are very nice (I use them and I am happy) and can be used while maintaining the code in github, so they are quickest way out of the current "emergency" but still I think that it would be wiser to move to GL, for the following reasons:

1. Freedom: GL is Free Software while GH is not. This should be important for a FOSS-based collaboration such as Tango.
2. Avoid vendor lock-in: GH is now owned by a company (Microsoft) with a not-so-great history of commitment to Free Software and of respect to privacy. I wouldn't be surprised if, like Travis just did, GH started trying to monetize its actions (or at least started to cut expenses at some point). In contrast, GL is owned by a company with a much better reputation in terms of free software friendliness. But even in the case that GL "became evil", their service could simply be forked (because of "1"), or we could simply self-host a GL instance without affecting our project configurations.
3. Homogenizing enviroments: most institutes in the Tango community already run their own GL instances locally. Having tango on GL would facilitate transferring projects from local GL instances to community-shared projects. Also, it would facilitate community learning and cooperating since we would be using the same interface and environment both internally and externally.
4. Improving CI: Continuous Integration in GL is very good and well-tested: it is definitely better than with travis+appveyor, and, IMHO, it is on par with GH actions (which is also very good). And in any case there is probably more knowledge on gitlab-ci than on GH actions within the Tango community members, which would simplify the migration from travis.
5. Packaging and dockerization: at ALBA we are using gitlab-ci for collaborative packaging. Debian also uses it for creating their official packages. Similarly, people at SKA are also basing their CI on GL. If tango was using gitlab-ci for its own CI as well, it would be very easy to automate package creation and/or dockerization as part of the tango projects CI.
6. Mail interface: Gitlab supports an email-based helpdesk (e.g. users could email to incoming+tango-controls/tango@incoming.gitlab.com and the emails are automatically converted to issues). This lowers the barrier for reporting issues and enables some automatizations.
7. Unlimited private projects. Gitlab.com allows to host unlimited private projects for free. This is not that relevant for a Free Software community like us, but still it can be interesting for some starting projects, or for some proprietary projects based on tango.

On the other hand, the main obvious reason in favor of moving to GH actions is that it does not require to move the code to GL.

Regarding this, I would like to point out that, IMHO, the extra effort of migrating the repos from GH to GL is not too much, because:

• migrating a full project from GH to GL is automated: using the automatic import button, the repo, the wiki, the issues, milestones, labels, etc are cloned in gitlab in a few minutes (I tested this with the taurus project).
• GL supports synchronizing with GH repos: the transition can be facilitated by leaving the GH repo as a synchronized mirror of the GL one for a while
• new accounts are not required: our users, who already have a GH account, don't even need to create a gitlab.com account (gitlab.com uses openID so if you want, you can log in directly with your GH account, or google or twitter...).

My view is that the little extra work in the short term is well worth compared to the mentioned advantages

Should the tango collaboration seriously consider this, I could offer my help with it.

[t-b]

We (as in byte physics) are using gitlab internally on-premise with a paid plan.

Yes github is now owned by microsoft. But gitlab is a startup and might also be tomorrow owned by $BigCompany. I think we should focus on getting tango better and avoid a migration like that.

Edit: Regarding the "Avoid vendor lock-in". This is a mere theoretical option to fork GL. It a huge pile of code developed in a very fast-pace. Without the orignal developers you can throw that only away.

[cpascual]

@t-b :

But gitlab is a startup and might also be tomorrow owned by $BigCompany.

See my point 2 regarding that: the key difference is that, because of GL being free software, even if GL is bought and/or stops being FOSS-friendly, it would be trivial for the Tango Collaboration to pay for a hosting (or use the infrastructure of one of its institutes) and host our own GL instance and continue without affecting the projects. You cannot do that with GH.

[cpascual]

This is a mere theoretical option to fork GL. It a huge pile of code developed in a very fast-pace. Without the orignal developers you can throw that only away.

That is definitely a good point, but I would not be as categorical... forks are not easy but they do happen and succeed in projects that have a large enough user base and whose owners stop being FOSS-friendly (see e.g. mariadb). And, in any case, the migration path to whatever new comes next is always easier from an open source solution than from a proprietary product (GH in this case)

[tjuerges]

Hands down, I would always choose Gitlab. They grant Open Source projects 50000 CI pipeline minutes and access to their top tier features that cost a lot of money otherwise if you do not host it yourself. After a very fast application process, ASTRON has been awarded a full license and we even host it now on our premises. Please see here: https://about.gitlab.com/solutions/open-source/join/

[t-b]

It is not about choosing, it is about migrating.

[andygotz]

Interesting discussion. I am a fervent supporter of OSS but I have difficulty seeing GL as good and GH as evil. They are different flavours of the same thing - git. I would not bet all my money on GL staying free for ever. If they are offering free services today they are probably counting on us to pay for them tomorrow. This is how startup companies work. Counting on a fork of GL when this happens is a possibility but not guaranteed as @t-b pointed out.

@cpascual offered good arguments on why to move but the main question to me is who is willing to host and setup a local GH instance for all the Tango projects. It would mean maintaining it at a level required by the developers i.e. high availability, updating of patches, sufficient disk space and platforms for running CI on all supported platforms. Personally we (ESRF) do not have the resources to offer this service now. That is why I am in favour of using GH in the meantime. The other option is to pay GL to host the Tango repositories or apply for a licence but then someone has to provide the infrastructure. Don't forget we are talking about ALL Tango repositories including hdbpp and related ones.

[cpascual]

I also would not count on GL to remain free (as in free beer) forever, but it is guaranteed to be free (as in freedom) because of their license (they can stop developing under the MIT license, but what they have already developed cannot be closed).

In contrast, GH is already non-free (as in freedom), and with Microsoft behind it, it does not seem unlikely that it may stop being free as in free beer as soon as they feel they can make money with such a move.

Regarding self-hosting GL (if the need arises): hosting a gitlab instance for something of the size of the Tango community (including all of it) is not specially demanding, IMHO. It is certainly a lot less demanding than what many of our institutes are currently doing with their own GL instances (thinking of ALBA or SKA or ESRF here). But in any case, it is something that the Tango Consortium could easily and cheaply outsource to any provider (if we ever need it).

And let me insist, it is not about GL=good vs GH=bad, but about getting out of a place that we cannot control before we invest too much time and make our tools too dependent on it. Plus also about using the same tools and workflows for the Collaboration than those we already use internally at the institutes.

And regarding the migration itself (as @t-b pointed it as "the" issue) I already mentioned that from the technical point of view it is quite easy (a lot easier than when we moved from sourceforge to GH). IMHO, the most difficult part of it is the non-technical aspect of agreeing on which day the GH projects should be made read-only (and communicating to all developers the new git url) ;-)

[tjuerges]

@t-b

It is not about choosing, it is about migrating.

Of course. That's why I wanted to add relevant information that could support making an informed decision.

@andygotz The free of charge licences are either Gold (Gitlab hosted) or Ultimate (self hosted). ASTRON just opted to self-host to make the LOFAR repository transition from SVN as seamless as possible. The only difference that could affect projects is that the Gitlab hosted projects are limited to 50000 CI minutes per month and naturally the self-hosted projects are not.

[mliszcz]

The only difference that could affect projects is that the Gitlab hosted projects are limited to 50000 CI minutes per month and naturally the self-hosted projects are not.

Both gitlab.com and github.com allow you to use your own, self-hosted runners if you need more CI time. I don't have any experience with GH but in GL adding a runner is trivial and all you need is a spare computer with Internet access (private IP is fine).

[bourtemb]

There is also the intermediate solution to trigger gitlab-ci from github: https://about.gitlab.com/solutions/github/

[cpascual]

trigger gitlab-ci from github

Yes, that option exists, but I do not think that it is practical: it requires a premium account and it does not integrate well with Pull Requests: https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/

[bourtemb]

Well, on the first link I've found, they said the following:

Who is GitLab CI/CD for GitHub for?
Open source projects

If you have a public, open source project on GitHub you can now take advantage of free CI/CD on GitLab.com. 
As part of our commitment to open source, we offer all public projects our highest tier features (Gold) for free. 
While other CI/CD vendors limit you to running a handful of concurrent jobs, 
GitLab.com gives open source projects hundreds of concurrent jobs with 50,000 free CI pipeline minutes

That seems contradictory with what is written in the documentation you pointed out. In any case, we still need to agree on a way forward because we had no time to discuss this during the kernel teleconf yesterday. And we need to do something before 31st December 2020.

[t-b]

I'm using CI/CD in gitlab for a github repo. It is okay, but not more. It also does not give detailed status information on per job basis but only a global good/bad.

[cpascual]

you are right @bourtemb : gitlab gives the "gold" account to Open Source Projects, so this would help to get more time for the transition. But, personally I would not use that as a permanent solution because the integration is not that good: there is what @t-b said, and also tests are not run when doing a PR from a forked project (which is the usual when using the fork-n-pull workflow).

[cpascual]

If you are considering the possibility of moving to gitlab.com, I propose the following: we can import a couple of GH repositories into the https://gitlab.com/tango-controls group (which was created 5 years ago when we evaluated GL as a destination from Sourceforge) to get a more direct feeling of the process.

I can do it, but I would need to be added as an admin of the selected repo(s) in order to be able to do the full import

[tjuerges]

tests are not run when doing a PR from a forked project (which is the usual when using the fork-n-pull workflow).

Is that not a configuration option with regard to how PR are handled?

[t-b]

Is that not a configuration option with regard to how PR are handled?

No. In gitlab this is an open issue, see https://gitlab.com/gitlab-org/gitlab/-/issues/5667.

[tjuerges]

No. In gitlab this is an open issue, see https://gitlab.com/gitlab-org/gitlab/-/issues/5667.

Bummer. Thanks for sharing @t-b .

[bourtemb]

Also, while we are adding more jobs, travis is reducing number of concurrent jobs. Maybe this is an opportunity to look for alternative. https://docs.travis-ci.com/user/migrate/open-source-repository-migration#q-will-there-be-lower-concurrency-for-free-accounts-on-travis-ciorg

[...] free and open-source .org accounts will have concurrency reduced from 5 to 4 concurrent jobs [...]

Well, the sentence quoted here is talking about open-source .org account (which means open-source account still on travis-ci.org, not yet migrated to travis-ci.com). Here are the whole sentences which say that the number of concurrent jobs is not changed on travis-ci.com. So after the migration to travis-ci.com, we'll still have 5 concurrent jobs:

As part of the shift of infrastructure from .org to .com and needing to make sure all users have equal access to resources, free and open-source .org accounts will have concurrency reduced from 5 to 4 concurrent jobs. Concurrent jobs have not changed on .com, so please consider migrating your repositories as soon as possible if this is an issue.

[bourtemb]

Personally, I would be in favour of migrating to travis-ci.com first because it really seems to be the easiest for now. The migration takes only a few seconds according to their documentation.

[mliszcz]

Well, the sentence quoted here is talking about open-source .org account (which means open-source account still on travis-ci.org, not yet migrated to travis-ci.com).

I misunderstood this, sorry! I though it meant that free .org accounts running on .com will have limited concurrency. I'll update the original post to prevent any further confusion.

[cpascual]

I would be in favour of migrating to travis-ci.com

but how does that affect to the limitation on CI minutes? If I am not mistaken, that (as oppossed to concurrency) is by far the main issue triggering the need of an alternative.

...or maybe I am missing something?

[bourtemb]

I would be in favour of migrating to travis-ci.com

but how does that affect to the limitation on CI minutes? If I am not mistaken, that (as oppossed to concurrency) is by far the main issue triggering the need of an alternative.

...or maybe I am missing something?

If I read correctly this blog post, Travis seems to do the same as Gitlab for Open Source projects:

We will be offering an allotment of OSS minutes that will be reviewed and allocated on a case by case basis. Should you want to apply for these credits please open a request with Travis CI support stating that you’d like to be considered for the OSS allotment. Please include:

Your account name and VCS provider (like travis-ci.com/github/[your account name] ) How many credits (build minutes) you’d like to request (should your run out of credits again you can repeat the process to request more or discuss a renewable amount)

[cpascual]

If I got it right, there is a big difference between how gitlab and travis treat the OSS projects:

• in GL they give a monthly budget of minutes (which is already generous but can even be increased on case-by case)
• Travis they just allocate an initial budget of minutes, and then you need to apply for more in a case-by-case.

So, the travis allotment is only useful for people wanting to test the service, but not for running on the long time.

[bourtemb]

@cpascual I understand your point. Indeed, there is a difference between having a monthly budget of minutes and having to request very often a renewal like in the Travis use case. I agree with you this is not convenient.

So, the travis allotment is only useful for people wanting to test the service, but not for running on the long time.

I think you're not fair with Travis on this point. The allotment are not only for testing the service. They always claimed they would stay free for OSS projects. https://blog.travis-ci.com/oss-announcement

You misunderstood my point of view as well when quoting me:

I would be in favour of migrating to travis-ci.com

You truncated my sentence. My opinion was:

I would be in favour of migrating to travis-ci.com first

I am still open to alternative. I was just suggesting to migrate to travis-ci.com as a first step to fix this emergency situation easily. We can still discuss the move to another service.

[cpascual]

Hi @bourtemb : my excuses if I came out as unjust to your comment . Nothing farther from my intentions!

I agree in that moving to travis.com can be done to gain some time, but I just wanted to remark that what Travis is offering from now on is not IMHO adequate for "production".

I do not think that stating this is being unfair: as I see it, Travis is 100% "morally" entitled to move to this new model and I am very grateful for what they already gave to the OSS ecosystem, but the fact is that the new model does not align well with our needs any more.

[bourtemb]

Hi @bourtemb : my excuses if I came out as unjust to your comment . Nothing farther from my intentions!

No worries @cpascual . I just wanted to be sure that my point of view was not misunderstood.

After reading the Travis community forum (https://travis-ci.community/t/org-com-migration-unexpectedly-comes-with-a-plan-change-for-oss-what-exactly-is-the-new-deal/10567/15) , I'm really starting to agree with you and being convinced we should actually even not try to migrate to travis-ci.com because we'll run out of credits immediately.

And there is no warranty that we will get an Open Source subscription validated because we might be considered as a sponsored project.

I agree in that moving to travis.com can be done to gain some time, but I just wanted to remark that what Travis is offering from now on is not IMHO adequate for "production".

I do not think that stating this is being unfair: as I see it, Travis is 100% "morally" entitled to move to this new model and I am very grateful for what they already gave to the OSS ecosystem, but the fact is that the new model does not align well with our needs any more.

After reading their community forum, I think you might be right. There are some contradictory points in the Travis communication which do not inspire trust.

[bourtemb]

If you are considering the possibility of moving to gitlab.com, I propose the following: we can import a couple of GH repositories into the https://gitlab.com/tango-controls group (which was created 5 years ago when we evaluated GL as a destination from Sourceforge) to get a more direct feeling of the process.

I can do it, but I would need to be added as an admin of the selected repo(s) in order to be able to do the full import

@cpascual , as an owner of tango-controls github organization, I think you are already all mighty.

[t-b]

Let's discuss that on thursday in the kernel meeting.

[andygotz]

FYI another open source project from the ESRF (silx.org) has not been granted OSS status because it is supported by an organisation and people working on this project are paid for by the organisation. This is the case for Tango too so we will not be eligible I think. Just wanted to inject this into the discussion this afternoon.

[lorenzopivetta]

Hi @andygotz , all, just checking, isn't the meeting tomorrow ? (e.g. Thursday Dec 3)

[bourtemb]

Hi @andygotz , all, just checking, isn't the meeting tomorrow ? (e.g. Thursday Dec 3)

Indeed, there is a meeting tomorrow at 14:00 CET to talk about #814 . Maybe we could take this opportunity to talk about that too?

[lorenzopivetta]

Hi @andygotz , all, just checking, isn't the meeting tomorrow ? (e.g. Thursday Dec 3)

Indeed, there is a meeting tomorrow at 14:00 CET to talk about #814 . Maybe we could take this opportunity to talk about that too?

That was my understanding reading @t-b comment, unless he's proposing next scheduled meeting on Dec 10. Not bad if we can take advantage tomorrow, however.

[t-b]

I was more thinking about the next scheduled meeting.

[cpascual]

So, will this be discussed today? or next Thursday?

[bourtemb]

So, will this be discussed today? or next Thursday?

@cpascual I propose to discuss this today too because something needs to be done before the end of this year and the holidays are approaching fast for many of us. For people not interested in the device level dynamic attributes support discussion, I will send a message on tango-controls kernel slack channel when we start discussing about this travis -ci.com issue.

[cpascual]

Hi, here are some notes on a quick test of migration of cpptango and pytango:

https://gitlab.com/-/snippets/2046070