search

DistributedScience / Distributed-CellProfiler

Run encapsulated docker containers with CellProfiler in the Amazon Web Services infrastructure.
https://distributedscience.github.io/Distributed-CellProfiler/
Other
39 stars 25 forks source link

Need to update queue code #133

Open bethac07 opened 2 years ago

bethac07 commented 2 years ago 
/home/ubuntu/.pyenv/versions/3.8.10/lib/python3.8/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: queue.amazonaws.com. In the next minor version this will be moved to sqs.us-east-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details
ErinWeisbart commented 2 years ago

I think we don't actually need to do anything?

My understanding of the botocore issue is that they are deprecating accepting endpoints given as {region}.{service}.{dnsSuffix} and will only accept {service}.{region}.{dnsSuffix}.

I don't think this applies to us as we are not explicitly naming the endpoint nor are we explicitly allowing domains anywhere. Instead, we use boto3 client sqs.list_queues() and boto3 resource sqs.get_queue_by_name().

bethac07 commented 2 years ago

Usually a future warning will only get popped if you're actually passing it the format that's about to be deprecated, so we probably want to check what we're currently getting - but yes, you would HOPE that in general if boto3 is handling it all, it will do so correctly. Might just mean a minimum version of boto3 and/or botocore.

ErinWeisbart commented 2 years ago

Where/when did the warning print for you?

I've got a run going now. I made my own DCP docker FROM cellprofiler/cellprofiler:4.2.4. I'm not noticing this warning anywhere but the queue I made defaulted to having Server-Side Encryption which is new...

bethac07 commented 2 years ago

A couple of different machines that I was launching DCP from. I can grab the pip freeze's next time I'm on. Since it's on the machine you launch from, rather than the Docker, the exact docker version shouldn't matter.

ErinWeisbart commented 2 years ago

Ah, right! The machine I just used has boto3==1.16.49 botocore==1.19.49.

Full pip freeze attrs==19.3.0 Automat==0.8.0 awscli==1.18.209 blinker==1.4 boto3==1.16.49 botocore==1.19.49 certifi==2019.11.28 chardet==3.0.4 click==7.1.2 cloud-init==22.2 colorama==0.4.3 command-not-found==0.3 configobj==5.0.6 constantly==15.1.0 cryptography==2.8 dbus-python==1.2.16 distro==1.4.0 distro-info===0.23ubuntu1 docutils==0.15.2 ec2-hibinit-agent==1.0.0 entrypoints==0.3 hibagent==1.0.1 httplib2==0.14.0 hyperlink==19.0.0 idna==2.8 importlib-metadata==1.5.0 incremental==16.10.1 Jinja2==2.10.1 jmespath==0.10.0 jsonpatch==1.22 jsonpointer==2.0 jsonschema==3.2.0 keyring==18.0.1 language-selector==0.1 launchpadlib==1.10.13 lazr.restfulclient==0.14.2 lazr.uri==1.0.3 MarkupSafe==1.1.0 more-itertools==4.2.0 netifaces==0.10.4 oauthlib==3.1.0 pexpect==4.6.0 pyasn1==0.4.8 pyasn1-modules==0.2.1 PyGObject==3.36.0 PyHamcrest==1.9.0 PyJWT==1.7.1 pymacaroons==0.13.0 PyNaCl==1.3.0 pyOpenSSL==19.0.0 pyrsistent==0.15.5 pyserial==3.4 python-apt==2.0.0+ubuntu0.20.4.3 python-dateutil==2.8.1 python-debian===0.1.36ubuntu1 PyYAML==5.3.1 requests==2.22.0 requests-unixsocket==0.2.0 rsa==4.5 s3transfer==0.3.3 SecretStorage==2.3.1 service-identity==18.1.0 simplejson==3.16.0 six==1.15.0 sos==4.0 ssh-import-id==5.10 systemd-python==234 tqdm==4.64.0 Twisted==18.9.0 ubuntu-advantage-tools==20.3 ufw==0.36 unattended-upgrades==0.1 urllib3==1.26.2 wadllib==1.3.3 zipp==1.0.0 zope.interface==4.7.1
ErinWeisbart commented 2 years ago

If I update to boto3==1.24.77 and botocore==1.27.77 I do get the warning.

My queue is also still defaulting to SSE though it looks like my machines are pulling jobs from the queue without a problem so changing back to our old default of Encryption Disabled may not be necessary...