Closed shcheklein closed 3 years ago
ERROR: failed to reproduce 'dvc.yaml': failed to run: python src/featurization.py data/prepared data/features, exited with 137
ondvc exp run --run-all
OOM?
Yep, this is OOM. It's not reliable, sometimes happens, sometimes not. This is the reason I want to start updating the Experiments with Tensorflow/MNIST first.
@shcheklein
After running
bin/run-in-container.py -c dvcorg/doc-katacoda:start-experiments -k execute ../katacoda-scenarios/get-started/06-experiments/*.md
I get the following. All other problems are Katacoda specific.
### RUNNING ###
$ dvc exp --help
###############
usage: dvc experiments [-h] [-q | -v]
{show,apply,diff,run,gc,branch,list,push,pull,remove}
...
Commands to run and compare experiments.
Documentation: <https://man.dvc.org/exp>
positional arguments:
{show,apply,diff,run,gc,branch,list,push,pull,remove}
Use `dvc experiments CMD --help` to display command-
specific help.
show Print experiments.
apply Apply the changes from an experiment to your
workspace.
diff Show changes between experiments in the DVC
repository.
run Reproduce complete or partial experiment pipelines.
gc Garbage collect unneeded experiments.
branch Promote an experiment to a Git branch.
list List local and remote experiments.
push Push a local experiment to a Git remote.
pull Pull an experiment from a Git remote.
remove Remove local experiments.
optional arguments:
-h, --help show this help message and exit
-q, --quiet Be quiet.
-v, --verbose Be verbose.
### RUNNING ###
$ dvc exp run --set-param featurize.max_features=1500 -S featurize.ngrams=2
###############
### RUNNING ###
$ git diff params.yaml
###############
diff --git a/params.yaml b/params.yaml
index 8f34d08..bd501e9 100644
--- a/params.yaml
+++ b/params.yaml
@@ -3,8 +3,8 @@ prepare:
seed: 20170428
featurize:
- max_features: 500
- ngrams: 1
+ max_features: 1500
+ ngrams: 2
train:
seed: 20170428
### RUNNING ###
$ dvc exp diff
###############
Path Param Value Change
params.yaml featurize.max_features 1500 1000
params.yaml featurize.ngrams 2 1
### RUNNING ###
$ dvc exp run --queue -n exp-1 -S train.n_est=50
###############
Queued experiment 'abce64c' for future execution.
### RUNNING ###
$ dvc exp run --queue -n exp-2 -S train.n_est=100
###############
Queued experiment '2fd7df5' for future execution.
### RUNNING ###
$ dvc exp run --queue -n exp-3 -S train.n_est=150
###############
Queued experiment '75f4350' for future execution.
### RUNNING ###
$ dvc exp run --queue -n exp-4 -S train.n_est=200
###############
Queued experiment '6f5c221' for future execution.
### RUNNING ###
$ dvc exp run --run-all
###############
Verifying data sources in stage: 'data/data.xml.dvc'
Verifying data sources in stage: 'data/data.xml.dvc'
Verifying data sources in stage: 'data/data.xml.dvc'
Verifying data sources in stage: 'data/data.xml.dvc'
Running stage 'prepare':
> python src/prepare.py data/data.xml
Running stage 'prepare':
> python src/prepare.py data/data.xml
Running stage 'prepare':
> python src/prepare.py data/data.xml
Running stage 'prepare':
> python src/prepare.py data/data.xml
### RUNNING ###
$ dvc exp show --no-timestamp --include-params train.n_est --no-pager
###############
โโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Experiment โ avg_prec โ roc_auc โ train.n_est โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ workspace โ 0.52048 โ 0.9032 โ 50 โ
โ katacoda-project โ 0.52048 โ 0.9032 โ 50 โ
โ โโโ *6f5c221 โ - โ - โ 200 โ
โ โโโ *75f4350 โ - โ - โ 150 โ
โ โโโ *2fd7df5 โ - โ - โ 100 โ
โ โโโ *abce64c โ - โ - โ 50 โ
โโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโ
### RUNNING ###
$ dvc exp apply exp-2
###############
### RUNNING ###
$ git add dvc.lock params.yaml prc.json roc.json scores.json
###############
### RUNNING ###
$ git commit -m "Preserve best Avg. Prec. experiment"
###############
[detached HEAD 06c52aa] Preserve best Avg. Prec. experiment
1 file changed, 2 insertions(+), 2 deletions(-)
### RUNNING ###
$ dvc exp show --no-timestamp --include-params train.n_est --no-pager
###############
โโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Experiment โ avg_prec โ roc_auc โ train.n_est โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ workspace โ 0.52048 โ 0.9032 โ 50 โ
โ 06c52aa โ 0.52048 โ 0.9032 โ 50 โ
โโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโ
### RUNNING ###
$ dvc exp show -n 2 --no-timestamp --include-params train.n_est --no-pager
###############
โโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Experiment โ avg_prec โ roc_auc โ train.n_est โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ workspace โ 0.52048 โ 0.9032 โ 50 โ
โ 06c52aa โ 0.52048 โ 0.9032 โ 50 โ
โ katacoda-project โ 0.52048 โ 0.9032 โ 50 โ
โ โโโ *6f5c221 โ - โ - โ 200 โ
โ โโโ *75f4350 โ - โ - โ 150 โ
โ โโโ *2fd7df5 โ - โ - โ 100 โ
โ โโโ *abce64c โ - โ - โ 50 โ
โโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโ
### RUNNING ###
$ dvc exp gc --workspace
###############
### RUNNING ###
$ dvc exp show -n 2 --no-timestamp --include-params train.n_est --no-pager
###############
โโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโณโโโโโโโโโโณโโโโโโโโโโโโโโ
โ Experiment โ avg_prec โ roc_auc โ train.n_est โ
โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ
โ workspace โ 0.52048 โ 0.9032 โ 50 โ
โ 06c52aa โ 0.52048 โ 0.9032 โ 50 โ
โ katacoda-project โ 0.52048 โ 0.9032 โ 50 โ
โ โโโ *6f5c221 โ - โ - โ 200 โ
โ โโโ *75f4350 โ - โ - โ 150 โ
โ โโโ *2fd7df5 โ - โ - โ 100 โ
โ โโโ *abce64c โ - โ - โ 50 โ
โโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโโโโโโ
I didn't close this review issue, feel free to update with a new review or close if the scenario is OK now. @shcheklein Thank you ๐๐ผ
Closing this for now. Thanks @iesahin . I'll create a new one if I see something else.
https://katacoda.com/dvc/courses/get-started/experiments
Issues:
ERROR: unexpected error - 'ascii' codec can't encode characters in position 462-464: ordinal not in range(128)
ondvc exp show
Step 3Update available 2.0.15 -> 2.0.17
for every command - we need to run an update before? Run CI builds all those containers regularly?ERROR: failed to reproduce 'dvc.yaml': failed to run: python src/featurization.py data/prepared data/features, exited with 137
ondvc exp run --run-all
OOM?๐ See Experiment Management for more information on DVC's approach.
- link is broken