search

Neuraxio / Neuraxle

The world's cleanest AutoML library ✨ - Do hyperparameter tuning with the right pipeline abstractions to write clean deep learning production pipelines. Let your pipeline steps have hyperparameter spaces. Design steps in your pipeline like components. Compatible with Scikit-Learn, TensorFlow, and most other libraries, frameworks and MLOps environments.
https://www.neuraxle.org/
Apache License 2.0
608 stars 62 forks source link

Feature: Use trial_hash instead of trial number when creating a log file. #509

Closed vincent-antaki closed 2 years ago

vincent-antaki commented 3 years ago

At the moment, when creating a log file for a trial, we use the trial number to in the log file path. This is done because the trial hash is dependent on the value of the hyperparameters and these value change during execution. This makes it harder to see the mapping between log file and run, as most information (such as generated image and pipeline dump) are stored using the trial_hash while the log is under the trial number. Furthermore this also has the following side effect that if, for some kind of reason, you run a second AutoML loop with the same HyperparamsJSONRepository, then the log already present would be overwritten (even though the rest of the trial information would still be ok).

I feel like the solution to this problem is simple : have the trial_hash be fixed by trial (instead of hyperparameter dependent) and use the trial_hash in the name of the log file instead of the trial number

Bonus : This would in part solve issue #493. It should be done by anyone which attempt to complete #504.

Note : I was under the impression I've logged this issue already but am unable to find it at the moment. So there might be a possible duplicate.

guillaume-chevalier commented 3 years ago

Related to #504 and to present #509 - should sort files by ID as well. See code in question in neuraxle/metaopt/auto_ml.py :

        trials = Trials()

        files = glob.glob(os.path.join(self.cache_folder, '*.json'))

        # sort by created date:
        def getmtimens(filename):
            return os.stat(filename).st_mtime_ns

        files.sort(key=getmtimens)

And remove the trial.save sleep that was used to space them out.

guillaume-chevalier commented 2 years ago

Closing this, as new trials won't have hashes anymore. New trials will now use a locking mechanism to be properly numbered without numbering collisions, in 0.7.0 and above.