missaugustina
commented
7 years ago Hi Augustina,
So I noticed you are using TravisCI for your testing. What things does it do really well for you?
I like
- the integration with GitHub pull requests
- the IRC hooks
- the perl-helpers support, so that we can define a matrix of Perls to build against. Not crucial for MetaCPAN but nice for CPAN modules
Is there anything that you wish it had that it doesn't?
I've been using Wercker for a personal project and I really like that it allows you to run your private repositories and also three concurrent builds. When we had a hackathon a few weeks ago it took really long to get feedback from Travis since there are no concurrent builds. The queue just got really backed up.
Also, the Wercker build pages are way easier to read. You don't have to do as much scrolling and it gives you a sense of how long your build will take. I haven't figured out how to run it with different Perl versions yet. To be honest, it took me longer to get set up with Wercker, but for me it was worth it. I probably would have just used Travis because that's what I know, but the entry level pricing for private builds wasn't even in the ballpark for my hobby project.
Wercker lets you auto-deploy to some environments. It would be really great if Travis would auto-deploy some MetaCPAN builds after they pass.
Actually, one thing I'd like to do is have a process which runs nightly in Travis, runs a "carton install" and then runs the test suite with all of the latest CPAN modules installed. If the run is successful, I'd like to have that build create a new GitHub pull request which a human could then review and merge in the morning. Currently this sort of thing isn't possible with Travis. It's not a deal breaker, but it would be nice to have.
Do you feel you have to work around anything in order to use TravisCI?
With the lack of concurrent builds I sometimes have to go in and cancel a build manually to speed up the process. Also, with a project like MetaCPAN that has many different repositories it's possible that a build from another repo will hold up a build from an unrelated repo. That's a real pain.
Additionally, even if TravisCI works really well for you, is there anything that you really don't like about using it?
It's really hard to find errors in the build output. The page is massive and hard to navigate.
I also noticed that you have several projects that seem to depend on each other. How do you currently test for instance that a change on a branch in the metacpan-api isn't going to break something in one of your metacpan clients?
When we test metacpan-web what we do is just run some live tests against a production instance of metacpan-api. It's not ideal, but it gets us some helpful results and it also lets us know if we're about to break production. So, it's actually a helpful kind of workaround.
Do you have any testing in addition to what you're running on TravisCI?
We have some "swat" tests which are run via cron. Not extensive, but helpful: https://github.com/metacpan/metacpan-api-test
I hope there's something helpful in there. :)
Reach out to Metacpan community.