Problem with dependabot accessing public github repositories to update Pipfile

[weiji14]

Hi there,

I've been getting this weird permissions issue a little over a month now (since mid April?) on pipenv VCS dependencies in my Pipfile which looks something like below:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
...
pygmt = {editable = true, ref = "0.0.1a0-16-g7004aa0", git = "https://github.com/GenericMappingTools/pygmt.git"}

[requires]
python_version = "3.6"

[pipenv]
allow_prereleases = true

It's started showing up in https://github.com/weiji14/deepbedmap/issues/137 asking for me to:

... provide additional git credentials in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'. If you use a custom token for the host github.com make sure it has read access to this repo, too.

which I couldn't quite understand as the pygmt package in question was hosted as a public repository. I created an access token but that didn't solve the problem, so I thought maybe it was an issue with GenericMappingTools/pygmt itself, so I switched to my personal fork at weiji14/pygmt, and then dependabot started complaining about another VCS dependency geopandas in https://github.com/weiji14/deepbedmap/issues/140 which is also publicly hosted.

That was when I realized it wasn't an issue with pygmt, but something with dependabot accessing github repository packages in general. I thought I resolved it in https://github.com/weiji14/deepbedmap/issues/142 by installing the dependabot-preview app on my personal pygmt fork, but the issue kept popping up (see https://github.com/weiji14/deepbedmap/issues/143 and https://github.com/weiji14/deepbedmap/issues/144).

Could you examine the logs and see what is going on, as I feel like I've exhausted every possibility on my side. Let me know if you need any additional information.

Thanks

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[rebelagentm]

👋 @weiji14, we will look into this. Are you still experiencing these issues?

[weiji14]

Hi there! I think it might have been because dependabot was confused with pygmt which had an invalid requirements.txt file. I've since fixed the file here in this PR and I'll try now to see if it works :crossed_fingers: It's taken me a couple of months of manually updating my dependencies to figure out what the problem might be :laughing:

The error message could definitely be a bit better though, since it's not really a permissions issue, but a problem with the package itself (not being to resolve its subdependencies). I'm not quite sure how to word it properly, but maybe check the error logs (I've just pushed the "Bump Now" button).

[weiji14]

Hmm, nope, I was wrong again. Dependabot still doesn't work with a proper requirements.txt file in pygmt :cry:

[rebelagentm]

Thanks for getting back to us! We'll take a look.

[feelepxyz]

@weiji14 we've run into a bit of a wall on this one, our updates for your repository are hitting our 5gb space limit and failing to complete 😢 sadly not sure if there's an easy work around with the way we currently do python/pip installs.

[weiji14]

Ah bugger, could you be more specific on what the 5gb limit is for? I could definitely try to cut down on some dependencies...

[feelepxyz]

@weiji14 it's hitting the disk space limit in our docker container, presumably the pip install is filling it up. Hard to tell exactly and a bit tricky to find out as we don't have much instrumentation on what's downloaded.

[rebelagentm]

@weiji14 If you do try cutting down on some of the dependencies, we can have another go at troubleshooting further.

[weiji14]

Just wondering if running pipenv lock --clear on your end might help? I found 1 small dependency I can cut out easily, and I'm not sure if that'll make a dent to bring it below 5GB.

[feelepxyz]

@weiji14 hm not sure, the install seems to fail half way through, I'm guess the lock --clear has to be run after the install has completed? We also start each job with a clean slate. Hopefully we can migrate over to GitHub Actions which has more space allowance per container.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[infin8x]

Closing this due to age and inactivity. If this scenario is critical to your workflow please feel free to comment.