search

visual-layer / fastdup

fastdup is a powerful free tool designed to rapidly extract valuable insights from your image & video datasets. Assisting you to increase your dataset images & labels quality and reduce your data operations costs at an unparalleled scale.
Other
1.51k stars 74 forks source link

[Bug]: RuntimeError: fastdup detected your are running an old version 1.60 (10 versions or more vs. the latest) please upgrade fastdup) #321

Closed ashrielbrian closed 2 months ago

ashrielbrian commented 2 months ago

What happened?

fastdup is great, but man - you guys should seriously consider removing this RuntimeError because it brings down production ML pipelines under very unassuming circumstances. Take this as an example:

  1. Following fastdup's advice to not pin a version (already sus here), we add into our Dockerfile pip install fastdup --no-deps.
  2. Everything works great, life continues, our ML pipelines run in production over time.
  3. No changes in our pipelines over a couple of months, so no rebuilds of the docker image.
  4. One fine day, our pipeline runs - and poof! RuntimeError - a container that was previously working with all the same dependencies, is suddenly no longer working because for whatever reason, fastdup decided to check whether the user imported version is within 10 versions of the latest (why?!) at runtime.

This has happened to our team multiple times already, and we've removed the pinned fastdup version. And now with this, it's a bit ridiculous if I'm honest.

Only after checking check_latest_version did I find:

if 'FASTDUP_PRODUCTION' in os.environ:
            return False

This needs to be surfaced up in your documentation. But still - add a warning, not raise a RuntimeError...

What did you expect to see?

What version of fastdup were you runnning on?

v1.60

What version of Python were you running on?

Python 3.9

Operating System

Ubuntu 20.04

Reproduction steps

No response

Relevant log output

No response

Attach a screenshot [Optional]

No response

Contact Details [Optional]

No response

dbickson commented 2 months ago

Hi @ashrielbrian Please define environment variable FASTDUP_PRODUCTION=1, this will skip the latest code check. We recommend to upgrade from time to time as fastdup is constantly improving including better performance and stability.

elad165 commented 1 week ago

Hi @ashrielbrian, thanks for this feedback. We have a few ideas on how we can better support you, would you be interested in having a short meeting?