Lots of people are getting dependency conflicts when installing pydbtools. This is because s3fs >= 0.5.0 adds aiobotocore, which is very picky about the version of botocore it works with.
This pull request pins s3fs below 0.5.0 to resolve these version conflicts. It also adds a standard gitignore.
There's an alternative to pinning s3fs: pinning botocore to 1.17.44. This would let us use more recent s3fs versions. However, I've pinned s3fs instead because:
aiobotocore is very restrictive about its botocore version - it only works with 1.17.44. Issues about this on their Github suggest this is unlikely to change
botocore is updated frequently (it's had 23 more updates since 1.17.44), so pinning it is likely to restrict us more than pinning s3fs
we've had no problems with using s3fs 0.4.2
avoiding the need to install aiobotocore means fewer dependencies
If this becomes a problem in the future, we could remove s3fs from pydbtools and use boto3 instead. This might be easier than navigating the version conflicts - but it seemed excessive to go for this change straight away.
Lots of people are getting dependency conflicts when installing pydbtools. This is because s3fs >= 0.5.0 adds aiobotocore, which is very picky about the version of botocore it works with.
This pull request pins s3fs below 0.5.0 to resolve these version conflicts. It also adds a standard gitignore.
There's an alternative to pinning s3fs: pinning botocore to 1.17.44. This would let us use more recent s3fs versions. However, I've pinned s3fs instead because:
If this becomes a problem in the future, we could remove s3fs from pydbtools and use boto3 instead. This might be easier than navigating the version conflicts - but it seemed excessive to go for this change straight away.