explosion / weasel

🦦 weasel: A small and easy workflow system
MIT License
63 stars 8 forks source link

Fix Azure storage for assets and remote storage #93

Open jtnicholl-cosairus opened 2 months ago

jtnicholl-cosairus commented 2 months ago

Though the documentation claims you can use any protocol that Smart Open supports, Azure storage accounts don't work. I wrote a quick fix to get it working.

Description

Smart Open allows providing transport parameters, which are optional, unless you're using Azure storage containers, then they are required and must include a client that has the connection string.

I made a quick fix to grab the connection string from the environment variable AZURE_STORAGE_CONNECTION_STRING if the URL is Azure. I know I probably didn't follow conventions correctly, but I needed to use Azure and that got it working.

Types of change

Small change to remote.py.

Checklist

jtnicholl-cosairus commented 2 months ago

So after this I moved on to remote storage with push and pull, and ran into another problem. Smart Open expects Azure URLs to start with azure://, but Cloudpathlib expects az://. So when Cloudpathlib sees azure:// it thinks it's a local file path and changes it to azure:/, with the added bonus on Windows that it becomes azure:\ instead (along with all the other slashes). This prevents Azure from working for remote storage even after the above fix. I pushed another commit to replace az:// with azure:// in remote.py before passing it to Smart Open, which feels a bit hacky but I don't know how else to get it working.