search

conda-forge / staged-recipes

A place to submit conda recipes before they become fully fledged conda-forge feedstocks
https://conda-forge.org
BSD 3-Clause "New" or "Revised" License
709 stars 4.96k forks source link

Travis CI API issues (backlogged recipes) #923

Closed jakirkham closed 8 years ago

jakirkham commented 8 years ago

We are waiting for conversion to go through. Travis CI seems quite slow to start this process. As that has gotten a couple pings, I'm opening an issue here to track status of this issue.

If you have PR on staged-recipes that you are working on actively, please try merging with master and removing any recipes from the directory recipes that you are not proposing. Commit those changes and push.

jakirkham commented 8 years ago

Seems PR ( https://github.com/conda-forge/staged-recipes/pull/757 ) needs some configuration changes before we go ahead. As a result, that is blocking a successful run on staged-recipes, but is not the cause of the long build delay. Am reverting the change with PR ( https://github.com/conda-forge/staged-recipes/pull/926 ), which should fix this issue.

jakirkham commented 8 years ago

Reopening until we have actually cleared the backlog. However, measures have been put in place to address it.

jakirkham commented 8 years ago

Well, there was an issue with one recipe/conda-build, which we resolved with this PR ( https://github.com/conda-forge/staged-recipes/pull/928 ). However, now we find ourselves rate-limited part way through the process. Will remove the completely converted recipes so as to lessen the load in masters and people's PRs.

jakirkham commented 8 years ago

We have converted some of the feedstocks to recipes completely. However, we were immediately rate-limited, which blocked the process from completing. I have gone back through and checked to see which ones completed successfully. Those that converted fine have been manually removed via this PR ( https://github.com/conda-forge/staged-recipes/pull/929 ). Hopefully, the rest will be resolved in the next conversion. If not, we will do more manually removals until they are cleared.

jakirkham commented 8 years ago

Appears one round of manual removals was not enough. We were rate limited again. This PR ( https://github.com/conda-forge/staged-recipes/pull/930 ) does some more manual removals. As there are only 4 recipes left, I'm hopeful we won't need to do this again. 🍀

jakirkham commented 8 years ago

We finished our GitHub rate limit when PR ( https://github.com/conda-forge/staged-recipes/pull/929 ) was merged. So, we need to wait ~20mins before restarting the remainder for feedstock conversion.

jakirkham commented 8 years ago

Restarting to clear the remainder.

jakirkham commented 8 years ago

Had some issues connecting to Travis CI's API. Maybe this comment is relevant. We should revisit this issue, @pelson. Traceback below.

CI Summary for conda-forge/python-avro-feedstock (can take ~30s):
Traceback (most recent call last):
  File "/Users/travis/miniconda/bin/conda-smithy", line 9, in <module>
    load_entry_point('conda-smithy==0.10.4', 'console_scripts', 'conda-smithy')()
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 228, in main
    args.subcommand_func(args)
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 156, in __call__
    ci_register.add_project_to_travis(owner, repo)
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/ci_register.py", line 147, in add_project_to_travis
    response.raise_for_status()
  File "/Users/travis/miniconda/lib/python3.5/site-packages/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.travis-ci.org/auth/github
Traceback (most recent call last):
  File ".CI/create_feedstocks.py", line 186, in <module>
    subprocess.check_call(['conda', 'smithy', 'register-ci', '--feedstock_directory', feedstock_dir] + owner_info)
  File "/Users/travis/miniconda/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['conda', 'smithy', 'register-ci', '--feedstock_directory', '/var/folders/gw/_2jq29095y7b__wtby9dg_5h0000gn/T/tmp3l9o1evk__feedstocks/python-avro-feedstock', '--organization', 'conda-forge']' returned non-zero exit status 1

Restarted...

jakirkham commented 8 years ago

Ran into the Travis CI API issue again. Have removed the affected recipe via this PR ( https://github.com/conda-forge/staged-recipes/pull/932 ) as it already went through conversion completely.

jakirkham commented 8 years ago

Yeah, this is not going away. Now we are hitting it with python-snappy. Unfortunately that one is not fully converted. So, we are stuck. I suppose we could remove and readd things in different orders to clear. However, I simply don't have the time to do that. Looks like it is time for a conda-smithy bugfix release.

CI Summary for conda-forge/python-snappy-feedstock (can take ~30s):
Traceback (most recent call last):
  File "/Users/travis/miniconda/bin/conda-smithy", line 9, in <module>
    load_entry_point('conda-smithy==0.10.4', 'console_scripts', 'conda-smithy')()
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 228, in main
    args.subcommand_func(args)
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 156, in __call__
    ci_register.add_project_to_travis(owner, repo)
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/ci_register.py", line 147, in add_project_to_travis
    response.raise_for_status()
  File "/Users/travis/miniconda/lib/python3.5/site-packages/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.travis-ci.org/auth/github
Traceback (most recent call last):
  File ".CI/create_feedstocks.py", line 186, in <module>
    subprocess.check_call(['conda', 'smithy', 'register-ci', '--feedstock_directory', feedstock_dir] + owner_info)
  File "/Users/travis/miniconda/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['conda', 'smithy', 'register-ci', '--feedstock_directory', '/var/folders/gw/_2jq29095y7b__wtby9dg_5h0000gn/T/tmpphm2uuxo__feedstocks/python-snappy-feedstock', '--organization', 'conda-forge']' returned non-zero exit status 1

cc @pelson

jakirkham commented 8 years ago

Maybe we can try this PR ( https://github.com/conda-forge/conda-smithy/pull/204 )?

pelson commented 8 years ago

We have drawn the conclusion, based on testing within conda-smithy, that the TravisCI API is currently broken and will be raising it with them. I do not believe we are in a position to merge anything to staged-recipes until this is resolved (or we cease registering with TravisCI).

jakirkham commented 8 years ago

Please read the above comment @conda-forge/core.

jakirkham commented 8 years ago

I'm going to try restarting Travis CI to see if it will behave better this time. It has been 3hrs since the last run it failed on.

jakirkham commented 8 years ago

Nope. Failed again with the same error.

Did Travis CI get back to you, @pelson?

jakirkham commented 8 years ago

Though it does appear that python-snappy completed feedstock conversion ~3hrs ago. Let's get that one out of here then. Maybe we will get lucky and the rest will clear. 🙏 👼 🍀

jakirkham commented 8 years ago

No such luck. We are still having feedstock conversion issues. It is just now we are having them with redis-py. At least there is one less recipe sitting in master.

pelson commented 8 years ago

So it has worked periodically? Even more weird.

jakirkham commented 8 years ago

Yeah, I really don't get it. This is one of the reasons I have been giving issue ( https://github.com/conda-forge/staged-recipes/issues/915 ) more thought of late and also this comment ( https://github.com/conda-forge/staged-recipes/pull/849#issuecomment-229942441 ).

For instance, last night we had so many recipes awaiting conversion that builds were taking up to the max time for simple PRs (e.g. pure Python packages with few dependencies and import tests). This actually slowed are ability to try to clear things. Partially as I was unwilling to straight cancel people's builds to do so (fearing outlash).

jakirkham commented 8 years ago

FWIW it seems these eventually did clear. 🎉 Though I'm not sure we are not just going to hit this again as soon as we merge. 😒

jakirkham commented 8 years ago

Added the aforementioned comment to this issue ( https://github.com/conda-forge/staged-recipes/issues/942 ).

jakirkham commented 8 years ago

Appears we are having the same problems again with Travis CI. Reopening...

jakirkham commented 8 years ago

Manually removing one recipe that was converted ( https://github.com/conda-forge/staged-recipes/pull/945 ).

jakirkham commented 8 years ago

After many retries of running Travis CI, it has finally cleared the remaining recipes.

jakirkham commented 8 years ago

Seems to be back. Reopening...

jakirkham commented 8 years ago

Problem has been temporarily resolved. The remaining recipes cleared. Closing...

jakirkham commented 8 years ago

It seems to be acting more reliably now. Not really sure what changed though. 😕

Also, should note that 1 or 2 recipes normally seem to clear ATM. Though many of the problems had previously IIRC included bigger batches of recipes. Maybe there is something funky with the number of repos we are creating at a time and syncing with Travis CI. Perhaps we need a timeout (or longer timeout) at some point in our process. Something to think about anyways.

jakirkham commented 8 years ago

Still have no idea what caused the previous issues, but things are currently rolling smoothly through staged-recipes. Not going to question it, just going to take the win.

jakirkham commented 8 years ago

Looks like this is back with pyferret. Going to reopen and try retriggering. Feel free to close if this fixes it.

cc @ocefpaf

jakirkham commented 8 years ago

Looks like the Travis CI API issue went away after the restart. :confused:

jakirkham commented 8 years ago

Appears botocore and postgresql got stuck thanks to the same Travis CI API issue. Going to restart to see if that fixes it. Feel free to close if it does and I have not.

jakirkham commented 8 years ago

Seems the last restart did not fix it. Trying again.

jakirkham commented 8 years ago

Appears the third time was a charm. 🍀 Closing until this bites us again. 🐊

jakirkham commented 8 years ago

Appears we have another Travis CI API issue w.r.t win_unicode_console. Re-opening and restarting.

cc @ocefpaf (for awareness)

jakirkham commented 8 years ago

I've added a fresh Travis CI issue ( https://github.com/travis-ci/travis-ci/issues/6284 ) and cc'd you on the ticket @pelson. Let's see if we can figure this out.

jakirkham commented 8 years ago

FYI first restart failed. Trying a second one. Though if that doesn't work, maybe someone else can retry later.

jakirkham commented 8 years ago

...and a third restart.

jakirkham commented 8 years ago

Going to try and clear the completely converted recipes (alabaster) with PR ( https://github.com/conda-forge/staged-recipes/pull/989 ).

jakirkham commented 8 years ago

Still failed because of Travis CI's API (just like the other failures mentioned here). Restarted...

jakirkham commented 8 years ago

This is a new one. Still an issue with the Travis CI API, but there is another issue coming out.

Repository created, please edit conda-forge.yml to configure the upload channels
and afterwards call 'conda smithy register-github'
fatal: Cannot update paths and switch to branch 'master' at the same time.
Did you intend to checkout 'upstream_with_token/master' which can not be resolved as commit?
Switched to a new branch 'master'
CI Summary for conda-forge/win_unicode_console-feedstock (can take ~30s):
Traceback (most recent call last):
  File "/Users/travis/miniconda/bin/conda-smithy", line 9, in <module>
    load_entry_point('conda-smithy==0.10.4', 'console_scripts', 'conda-smithy')()
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 228, in main
    args.subcommand_func(args)
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/cli.py", line 156, in __call__
    ci_register.add_project_to_travis(owner, repo)
  File "/Users/travis/miniconda/lib/python3.5/site-packages/conda_smithy/ci_register.py", line 147, in add_project_to_travis
    response.raise_for_status()
  File "/Users/travis/miniconda/lib/python3.5/site-packages/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.travis-ci.org/auth/github
Traceback (most recent call last):
  File ".CI/create_feedstocks.py", line 186, in <module>
    subprocess.check_call(['conda', 'smithy', 'register-ci', '--feedstock_directory', feedstock_dir] + owner_info)
  File "/Users/travis/miniconda/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['conda', 'smithy', 'register-ci', '--feedstock_directory', '/var/folders/gw/_2jq29095y7b__wtby9dg_5h0000gn/T/tmpznykboxq__feedstocks/win_unicode_console-feedstock', '--organization', 'conda-forge']' returned non-zero exit status 1
jakirkham commented 8 years ago

Restarted it several times, which I did not record here as it was getting to noisy. Suffice it to say it took many tries, but it appears to now have cleared.

jakirkham commented 8 years ago

Happening again with setuptools. Trying to restart.

jakirkham commented 8 years ago

Tried a few restarts. No luck yet. 😞

jakirkham commented 8 years ago

So, I guess someone restarted it and it cleared. However we now have a new recipe stuck for the same reason. Have restarted it to see if we can clear it.

jakirkham commented 8 years ago

Yeah, that's pretty vague. setuptools cleared thanks to an assist from @pelson. However, bob.bio.face and then soon after cliff are stuck due to the same Travis CI API issue we have seen before.

jakirkham commented 8 years ago

Ooph. So, the last restart did manage to connect to Travis CI once and then failed on a subsequent connection. However, it was still using conda-smithy version 1.0.0. I have checked to see that conda-smithy version 1.0.1 builds are now available for Linux (which they are) and restarted Travis CI. Maybe this next restart will carry them through. 🍀

jakirkham commented 8 years ago

Seems this is functioning again for the moment.

jakirkham commented 8 years ago

Appears this issue has resurfaced.

jakirkham commented 8 years ago

Restarting current build on master to see if that will fix it.

jakirkham commented 8 years ago

So, this was fixed for a bit, but is now back with hachoir-core and hyperspy. Going to try restarting.