Why are some fetches failing?

[pelson]

conda-forge-maint> heroku run python conda-forge.github.io/scripts/lint_feedstocks.py --feedstocks-dir=./feedstocks/
Running python conda-forge.github.io/scripts/lint_feedstocks.py --feedstocks-dir=./feedstocks/ on conda-forge-maint... up, run.6285
Failed to fetch origin from https://github.com/conda-forge/bob.ip.facedetect-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/fortran-magic-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/django-redis-cache-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/glog-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/epic2cf-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/djangorestframework-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/geojson-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/feedgenerator-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/hdf4-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/geolinks-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/esmpy-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/gridutils-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/giflib-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/gmp-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/gdal-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/haversine-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/eofs-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/helper-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/hdfeos2-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/glibmm-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/libiconv-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/jq-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/libpng-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/glueviz-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/mkdocs-bootswatch-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/libmo_unpack-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/g2clib-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/mo_pack-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/megaman-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/libatomic_ops-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/hdfeos5-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/obvious-ci-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/mingwpy-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/luigi-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/libsvm-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/jsonschema-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/m4-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/miktex-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/google-apputils-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/ncurses-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/libuuid-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/megaman-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/libblitz-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/mplleaflet-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/isodate-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/netcdf4-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/munch-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/nco-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/mkdocs-bootstrap-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/pyamg-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pyface-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/pytest-mpl-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/nose_parameterized-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/pygrib-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/ptyprocess-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pint-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/python-dateutil-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/python-hdfs-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/pystache-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/python-drmaa-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pydicom-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pyopengl-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/python-ternary-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pypandoc-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pygrib-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pynio-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pygdp-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/python-leveldb-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/requests-toolbelt-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/qimage2ndarray-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/python-primesieve-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/rasterio-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/simplekml-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/siphon-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/shapely-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/suds-jurko-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/vlfeat-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/qutip-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pynco-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/vincent-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/regex-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/urwid-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/spyne-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/seawater-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/tabulate-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pythreejs-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/traitlets-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pyscaffold-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/thrift-cpp-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/rejected-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/trmm_rsl-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/vigra-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/vcrpy-feedstock.git.
Failed to fetch origin from https://github.com/conda-forge/termcolor-feedstock.git.
Failed to fetch upstream from https://github.com/conda-forge/pyresample-feedstock.git.
Collecting list of conda-forge repos...
Checking munch-feedstock
Checking nbformat-feedstock
Checking jsonschema-feedstock

This is consistently failing, yet the ones that fail are random (and by no means all of them). The fetch is being done in parallel, and it may be that there are simply http errors coming from the parallelism...

[jakirkham]

Do we need to pull down all of them? In particular, I'm thinking about re-rendering where we only pick 5 at a time.

[pelson]

In particular, I'm thinking about re-rendering where we only pick 5 at a time.

You are right for the heroku usecase - we could probably speed this up and simplify matters considerably by pulling on demand.

[pelson]

I changed the code to fetch only for the feedstock it is currently looking at as suggested. (see https://github.com/conda-forge/conda-smithy/pull/269).

Incidentally, the reason this was failing was due to a massive amount of parallelism being used for fetching, and getting heroku R14 errors such as:

2016-08-26T06:24:39.628631+00:00 heroku[run.2687]: Process running mem=549M(106.8%)
2016-08-26T06:24:39.628764+00:00 heroku[run.2687]: Error R14 (Memory quota exceeded)

[jakirkham]

That is a truly terrifying error. Glad to hear it was easily fixed though. 😄