mystic-ai / pipeline

Pipeline is an open source python SDK for building AI/ML workflows
https://www.mystic.ai
Apache License 2.0
117 stars 21 forks source link

construct url-safe url when downloading file onto container #453

Closed plutopulp closed 2 months ago

plutopulp commented 2 months ago

When downloading a file onto the container, we safe-encode the url before downloading the file, e.g. white spaces and other special chars will be encoded etc.. Also, to ensure backward compatibility (for pipelines using older pipeline-ai versions) and to encode any file_urls at the API entry level, we update the RunInput schema to recursively encode any file_url nested within. Note also we ensure that urls are not encoded multiple times.

linear[bot] commented 2 months ago

PC-1104 Files with spaces fail

plutopulp commented 2 months ago

Looks sensible to me. I have to admit, I don't fully understand the whole flow here but what you've done looks correct and like the fact that you've added tests to cover all the different permutations. I wasn't sure whether calling encode_url which has a pydantic decorator on it would work outside of pydantic validation but it obviously must do :)

Thanks for fixing this 👍

Yeah no worries 👍. I wasn't sure either about calling the pydantic decorated method from outside either, but it worked. Considered moving the encoding logic to a utility function defined outside the pydantic model, but probably fine as is.