search

coolneighbors / unWISE-verse

Integrated flipbook / panoptes pipeline
MIT License
2 stars 0 forks source link

Multithreading for AWS requests #7

Closed EdenSchapera closed 2 years ago

EdenSchapera commented 2 years ago

Implement multithreading for AWS requests to speed up downloading

ameisner commented 2 years ago

I'm concerned that this could get you banned/flagged as a DDoS attack? We should ask Dan about this.

EdenSchapera commented 2 years ago

Good thought, hadn't considered that. We should confirm before implementing this.

EdenSchapera commented 2 years ago

Sort of completed --- Could probably add additional multithreading for image postprocessing. But AWS side is handled

EdenSchapera commented 2 years ago

Closed. Added other image manipulation tasks to parallel processes. Working as intended. Annoying libmagic message, though.

austinh2001 commented 2 years ago

I was able to find a source for why the ligmagic message is there and, if you have another package, it will go away:

"This is a warning and affects you only if you are interested in working with media other than images ( ie video and audio etc) To fix it you need to install a different python_magic - if you have a 32 bit version of Python on a Windows 10 machine the version you need is python-magic-bin, and if you somehow have the 64 bit version of Python then you need python-magic-win64. Note the default download for Python is the 32 bit version and you would have to go looking in the alternate download list to find the 64 bit version as of the last time I checked - about 3 weeks ago. Once the new version is installed uninstall the original python-magic."

Source: https://www.zooniverse.org/talk/18/1447137?comment=2451603&page=1

For me, I installed the python-magic-bin package, ran PipelineStart.py, and the message was no longer there!