search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.68k stars 654 forks source link

conda environment usage when executing experiments #216

Open tomeramit opened 4 years ago

tomeramit commented 4 years ago

Hi,

I understood that trains support the use of conda (my need is to be able to launch experiments with different conda environments - e.g. launch train 1 with env 1 and then launch train 2 with env 2)

is it possible? (i'm adding the log for my training, when I tried to use conda, but conda doesnt install packages)

Thanks! Tomer

2020-10-07T10:46:14.052Z lv-beast:gpu0,1 INFO task 4ed39a8c42a64d5b8ac36d84c1064ce1 pulled from ca9acd7a32df49e5b4ea41760e252be7 by worker lv-beast:gpu0,1

2020-10-07T10:46:19.096Z lv-beast:gpu0,1 DEBUG Current configuration (trains_agent v0.16.1, location: /tmp/.trains_agent.pmazbr47.cfg):

sdk.storage.cache.default_base_dir = ~/.trains/cache sdk.storage.cache.size.min_free_bytes = 10GB sdk.storage.direct_access.0.url = file://* sdk.metrics.file_history_size = 100 sdk.metrics.matplotlib_untitled_history_size = 100 sdk.metrics.images.format = JPEG sdk.metrics.images.quality = 87 sdk.metrics.images.subsampling = 0 sdk.metrics.tensorboard_single_series_per_graph = false sdk.network.metrics.file_upload_threads = 4 sdk.network.metrics.file_upload_starvation_warning_sec = 120 sdk.network.iteration.max_retries_on_server_error = 5 sdk.network.iteration.retry_backoff_factor_sec = 10 sdk.aws.s3.key = sdk.aws.s3.region = sdk.aws.boto3.pool_connections = 512 sdk.aws.boto3.max_multipart_concurrency = 16 sdk.log.null_log_propagate = false sdk.log.task_log_buffer_capacity = 66 sdk.log.disable_urllib3_info = true sdk.development.task_reuse_time_window_in_hours = 72.0 sdk.development.vcs_repo_detect_async = true sdk.development.store_uncommitted_code_diff = true sdk.development.support_stopping = true sdk.development.default_output_uri = sdk.development.force_analyze_entire_repo = false sdk.development.suppress_update_message = false sdk.development.detect_with_pip_freeze = false sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true sdk.development.worker.report_global_mem_used = false agent.worker_id = lv-beast:gpu0,1 agent.worker_name = lv-beast agent.force_git_ssh_protocol = false agent.python_binary = agent.package_manager.type = conda agent.package_manager.pip_version = <20.2 agent.package_manager.system_site_packages = false agent.package_manager.force_upgrade = false agent.package_manager.conda_channels.0 = simpleitk agent.package_manager.conda_channels.1 = pytorch agent.package_manager.conda_channels.2 = astra-toolbox/label/dev agent.package_manager.conda_channels.3 = conda-forge agent.package_manager.conda_channels.4 = defaults agent.package_manager.torch_nightly = false agent.venvs_dir = /home/lv-beast/.trains/venvs-builds.1 agent.vcs_cache.enabled = true agent.vcs_cache.path = /home/lv-beast/.trains/vcs-cache.1 agent.venv_update.enabled = true agent.pip_download_cache.enabled = true agent.pip_download_cache.path = /home/lv-beast/.trains/pip-download-cache agent.translate_ssh = true agent.reload_config = false agent.docker_pip_cache = /home/lv-beast/.trains/pip-cache agent.docker_apt_cache = /home/lv-beast/.trains/apt-cache.1 agent.docker_force_pull = false agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04 agent.git_user = tomer.amit agent.cuda_version = 101 agent.cudnn_version = 70 agent.default_python = 3.7 api.version = 1.5 api.verify_certificate = true api.default_version = 1.5 api.http.max_req_size = 15728640 api.http.retries.total = 240 api.http.retries.connect = 240 api.http.retries.read = 240 api.http.retries.redirect = 240 api.http.retries.status = 240 api.http.retries.backoff_factor = 1.0 api.http.retries.backoff_max = 120.0 api.http.wait_on_maintenance_forever = true api.http.pool_maxsize = 512 api.http.pool_connections = 512 Executing task id [4ed39a8c42a64d5b8ac36d84c1064ce1]: branch = feature/add_amp version_num = afc213dfbc4fb320f7b314fd374c0ba13326d04f tag = entry_point = train_from_scratch_slices.py working_dir = LvObjects/Lv3D/TomoGan/scripts Executing Conda: /home/lv-beast/miniconda3/condabin/conda env remove -p /home/lv-beast/.trains/venvs-builds.1/3.7 --quiet --json

2020-10-07T10:46:49.172Z lv-beast:gpu0,1 DEBUG Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults cudatoolkit=10.1 --quiet --json

2020-10-07T10:47:14.235Z lv-beast:gpu0,1 DEBUG Pass Using cached repository in "/home/lv-beast/.trains/vcs-cache.1/MachineLearning.19ae66b07920a67ff810b2ac379f9292/MachineLearning" Fetching submodule LungVision Fetching submodule LungVision

2020-10-07T10:47:19.268Z lv-beast:gpu0,1 DEBUG Note: switching to 'afc213dfbc4fb320f7b314fd374c0ba13326d04f'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at afc213d Merge branch 'feature/sync_between_ml_to_tomo_integration' into feature/add_amp

2020-10-07T10:47:34.313Z lv-beast:gpu0,1 DEBUG type: git branch: HEAD commit: afc213dfbc4fb320f7b314fd374c0ba13326d04f root: /home/lv-beast/.trains/venvs-builds.1/3.7/task_repository/MachineLearning Applying uncommitted changes Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults 'pip<20.2' --quiet --json

2020-10-07T10:48:04.384Z lv-beast:gpu0,1 DEBUG Pass Executing Conda: /home/lv-beast/miniconda3/condabin/conda list --json -p /home/lv-beast/.trains/venvs-builds.1/3.7 Running task id [4ed39a8c42a64d5b8ac36d84c1064ce1]: [LvObjects/Lv3D/TomoGan/scripts]$ source /home/lv-beast/miniconda3/etc/profile.d/conda.sh && conda activate /home/lv-beast/.trains/venvs-builds.1/3.7 && /home/lv-beast/.trains/venvs-builds.1/3.7/bin/python -u train_from_scratch_slices.py Summary - installed python packages: conda:

jkhenning commented 4 years ago

Hi @tomeramit ,

It seems execution fails since the log_utils package is not installed:

tomeramit commented 4 years ago

Hi @jkhenning, thanks for the quick answer!

log utils is a PyPI package (https://pypi.org/project/log-utils/) I'm using git to launch the code (and the cloning of the code though git seems fine)

image you are correct, somehow the installed packages is empty, what can cause that?

jkhenning commented 4 years ago

Did you run your code manually to create the experiment? Since you have a direct import from this package, trains should have detected it and placed it in the "installed packages" section. Did you clone the experiment and possibly edited it manually?

jkhenning commented 4 years ago

BTW - you can always edit this section before enqueueing the experiment and add log_utils yourself - just to see if it works 🙂

tomeramit commented 4 years ago

I think that I run it manually, in order to add an experiment to the queue, I launch it on my comp, then go to the ui, restart and add it to the relevant queue, is there another/better way to do it? (I did not clone it)

jkhenning commented 4 years ago

@tomeramit might it be that during this process you've somehow cleared the "installed packages" section?

The recommended course of action is to run your code locally in order to create an experiment, and than go to the UI and clone the experiment (and not reset your original experiment) - the cloned experiment can then be added to the queue (using Enqueue)

tomeramit commented 4 years ago

I did it and it's still empty, can you tell me what fields to set in the config file (or send me and example) in order to do it? maybe something there is wrong?

tomeramit commented 4 years ago

I don't know how to clear the installed packages, so I guess it's not that

jkhenning commented 4 years ago

Can you try to do the recommended flow and see if it works for you?

tomeramit commented 4 years ago

yes, this is on the cloned one image

tomeramit commented 4 years ago

it seems like if I wait few seconds the "installed packages" appear, but the problem is that one (at least) of the packages (that appear when I do "conda list" for example) is missing from the list of "installed packages", for example astra package, although I added the channel: agent.package_manager.conda_channels.2 = astra-toolbox/label/dev (I have this package installed on my environment)

adding the logs, thanks for the help!

2020-10-11T08:39:50.608Z lv-beast:gpu0,1 INFO task 12dec4fc0c5a44ea8f89e8df293623ef pulled from ca9acd7a32df49e5b4ea41760e252be7 by worker lv-beast:gpu0,1

2020-10-11T08:39:55.640Z lv-beast:gpu0,1 DEBUG Current configuration (trains_agent v0.16.1, location: /tmp/.trains_agent.rpj1dtfw.cfg):

sdk.storage.cache.default_base_dir = ~/.trains/cache sdk.storage.cache.size.min_free_bytes = 10GB sdk.storage.direct_access.0.url = file://* sdk.metrics.file_history_size = 100 sdk.metrics.matplotlib_untitled_history_size = 100 sdk.metrics.images.format = JPEG sdk.metrics.images.quality = 87 sdk.metrics.images.subsampling = 0 sdk.metrics.tensorboard_single_series_per_graph = false sdk.network.metrics.file_upload_threads = 4 sdk.network.metrics.file_upload_starvation_warning_sec = 120 sdk.network.iteration.max_retries_on_server_error = 5 sdk.network.iteration.retry_backoff_factor_sec = 10 sdk.aws.s3.key = sdk.aws.s3.region = sdk.aws.boto3.pool_connections = 512 sdk.aws.boto3.max_multipart_concurrency = 16 sdk.log.null_log_propagate = false sdk.log.task_log_buffer_capacity = 66 sdk.log.disable_urllib3_info = true sdk.development.task_reuse_time_window_in_hours = 72.0 sdk.development.vcs_repo_detect_async = true sdk.development.store_uncommitted_code_diff = true sdk.development.support_stopping = true sdk.development.default_output_uri = sdk.development.force_analyze_entire_repo = false sdk.development.suppress_update_message = false sdk.development.detect_with_pip_freeze = false sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true sdk.development.worker.report_global_mem_used = false agent.worker_id = lv-beast:gpu0,1 agent.worker_name = lv-beast agent.force_git_ssh_protocol = false agent.python_binary = agent.package_manager.type = conda agent.package_manager.pip_version = <20.2 agent.package_manager.system_site_packages = true agent.package_manager.force_upgrade = false agent.package_manager.conda_channels.0 = simpleitk agent.package_manager.conda_channels.1 = pytorch agent.package_manager.conda_channels.2 = astra-toolbox/label/dev agent.package_manager.conda_channels.3 = conda-forge agent.package_manager.conda_channels.4 = defaults agent.package_manager.torch_nightly = false agent.venvs_dir = /home/lv-beast/.trains/venvs-builds agent.vcs_cache.enabled = true agent.vcs_cache.path = /home/lv-beast/.trains/vcs-cache agent.venv_update.enabled = false agent.pip_download_cache.enabled = true agent.pip_download_cache.path = /home/lv-beast/.trains/pip-download-cache agent.translate_ssh = true agent.reload_config = false agent.docker_pip_cache = /home/lv-beast/.trains/pip-cache agent.docker_apt_cache = /home/lv-beast/.trains/apt-cache agent.docker_force_pull = false agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04 agent.git_user = tomer.amit agent.cuda_version = 101 agent.cudnn_version = 70 agent.default_python = 3.7 api.version = 1.5 api.verify_certificate = true api.default_version = 1.5 api.http.max_req_size = 15728640 api.http.retries.total = 240 api.http.retries.connect = 240 api.http.retries.read = 240 api.http.retries.redirect = 240 api.http.retries.status = 240 api.http.retries.backoff_factor = 1.0 api.http.retries.backoff_max = 120.0 api.http.wait_on_maintenance_forever = true api.http.pool_maxsize = 512 api.http.pool_connections = 512 api.api_server = http://192.168.2.94:8008 api.web_server = http://192.168.2.94:8080 api.files_server = http://192.168.2.94:8081 api.credentials.access_key = IQ4BUOO3Q27ZPICO096M api.host = http://192.168.2.94:8008 Executing task id [12dec4fc0c5a44ea8f89e8df293623ef]: repository = https://bodyvisionmedical.visualstudio.com/DefaultCollection/LungVision/_git/MachineLearning branch = feature/add_amp version_num = 2cbc6f825800ccd7712a368cad22bae59566dd5e tag = entry_point = train_from_scratch_slices.py working_dir = LvObjects/Lv3D/TomoGan/scripts Executing Conda: /home/lv-beast/miniconda3/condabin/conda env remove -p /home/lv-beast/.trains/venvs-builds/3.7 --quiet --json Remove all packages in environment /home/lv-beast/.trains/venvs-builds/3.7:

2020-10-11T08:40:25.716Z lv-beast:gpu0,1 DEBUG Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults cudatoolkit=10.1 --quiet --json

2020-10-11T08:40:50.788Z lv-beast:gpu0,1 DEBUG Pass Using cached repository in "/home/lv-beast/.trains/vcs-cache/MachineLearning.6b0aaffa69d428f2574a1d5068d3ee6b/MachineLearning" From https://bodyvisionmedical.visualstudio.com/DefaultCollection/LungVision/_git/MachineLearning ed4d2b6..80f10a3 develop -> origin/develop

From https://bodyvisionmedical.visualstudio.com/DefaultCollection/_git/LungVision

2020-10-11T08:41:20.898Z lv-beast:gpu0,1 DEBUG Note: switching to '2cbc6f825800ccd7712a368cad22bae59566dd5e'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 2cbc6f8 add envs

2020-10-11T08:41:35.945Z lv-beast:gpu0,1 DEBUG type: git url: https://bodyvisionmedical.visualstudio.com/DefaultCollection/LungVision/_git/MachineLearning branch: HEAD commit: 2cbc6f825800ccd7712a368cad22bae59566dd5e root: /home/lv-beast/.trains/venvs-builds/3.7/task_repository/MachineLearning Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults 'pip<20.2' --quiet --json

2020-10-11T08:42:06.016Z lv-beast:gpu0,1 DEBUG Pass Conda: Trying to install requirements: ['certifi~=2020.6.20', 'chardet~=3.0.4', 'cryptography~=3.1.1', 'dask~=2.30.0', 'google-auth~=1.22.0', 'h5py~=2.10.0', 'imgaug~=0.4.0', 'joblib~=0.17.0', 'numba~=0.51.2', 'numpy~=1.19.1', 'packaging~=20.4', 'pandas~=1.1.3', 'Pillow~=7.2.0', 'pip~=20.2.3', 'plotly~=4.11.0', 'psutil~=5.7.2', 'pyclipper~=1.1.0', 'pytorch~=1.6.0', 'requests~=2.24.0', 'scikit-image~=0.16.2', 'scikit-learn~=0.23.2', 'scipy~=1.5.2', 'SimpleITK~=2.0.0', 'tensorboard~=2.3.0', 'torchvision~=0.7.0', 'urllib3~=1.25.10', 'graphviz', 'python-graphviz', 'kiwisolver'] Executing Conda: /home/lv-beast/miniconda3/condabin/conda env update -p /home/lv-beast/.trains/venvs-builds/3.7 --file /tmp/conda_env9ae7mze9.yml --quiet --json

2020-10-11T08:43:31.184Z lv-beast:gpu0,1 DEBUG Pass Conda: Installing requirements: step 2 - using pip: ['git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization', 'colorama==0.4.3', 'colorlog==4.4.0', 'jsonpickle==1.4.1', 'log_utils==0.3.4', 'matplotlib==3.3.2', 'tqdm==4.50.2', 'trains==0.16.2', 'vtk==8.1.2'] bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Processing /home/lv-beast/.cache/pip/wheels/c8/99/0d/0c61f3ff8d600f65c5557f941cfd316d0fc406f765223338b0/pyserialization-0.1.1-py3-none-any.whl Collecting colorama==0.4.3 Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB) Collecting colorlog==4.4.0 Using cached colorlog-4.4.0-py2.py3-none-any.whl (14 kB) Collecting jsonpickle==1.4.1 Using cached jsonpickle-1.4.1-py2.py3-none-any.whl (36 kB) Collecting log_utils==0.3.4 Using cached log_utils-0.3.4-py3-none-any.whl (11 kB) Requirement already satisfied: matplotlib==3.3.2 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqsoi878ysp.txt (line 6)) (3.3.2) Collecting tqdm==4.50.2 Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB) Collecting trains==0.16.2 Using cached trains-0.16.2-py2.py3-none-any.whl (790 kB) Collecting vtk==8.1.2 Using cached vtk-8.1.2-cp37-cp37m-manylinux1_x86_64.whl (48.9 MB) Requirement already satisfied: numpy in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from pyserialization->-r /tmp/cached-reqsoi878ysp.txt (line 1)) (1.19.2) Requirement already satisfied: importlib-metadata in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from jsonpickle==1.4.1->-r /tmp/cached-reqsoi878ysp.txt (line 4)) (2.0.0) Requirement already satisfied: python-dateutil>=2.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsoi878ysp.txt (line 6)) (2.8.1) Requirement already satisfied: kiwisolver>=1.0.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsoi878ysp.txt (line 6)) (1.2.0) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsoi878ysp.txt (line 6)) (2.4.7) Requirement already satisfied: cycler>=0.10 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsoi878ysp.txt (line 6)) (0.10.0) Requirement already satisfied: pillow>=6.2.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsoi878ysp.txt (line 6)) (7.2.0) Requirement already satisfied: certifi>=2020.06.20 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsoi878ysp.txt (line 6)) (2020.6.20) Requirement already satisfied: psutil>=3.4.2 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (5.7.2) Requirement already satisfied: plotly>=3.9.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (4.11.0) Requirement already satisfied: pyjwt>=1.6.4 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (1.7.1) Requirement already satisfied: six>=1.11.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (1.15.0) Collecting requests-file>=1.4.2 Using cached requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB) Requirement already satisfied: requests>=2.20.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (2.24.0) Collecting furl>=2.0.0 Using cached furl-2.1.0-py2.py3-none-any.whl (20 kB) Collecting pathlib2>=2.3.0 Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB) Requirement already satisfied: PyYAML>=3.12 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (5.3.1)

2020-10-11T08:43:36.204Z lv-beast:gpu0,1 DEBUG Collecting humanfriendly>=2.1 Using cached humanfriendly-8.2-py2.py3-none-any.whl (86 kB) Requirement already satisfied: urllib3>=1.21.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (1.25.10) Collecting jsonschema>=2.6.0 Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB) Requirement already satisfied: attrs>=18.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (20.2.0) Collecting funcsigs>=1.0 Using cached funcsigs-1.0.2-py2.py3-none-any.whl (17 kB) Processing /home/lv-beast/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from importlib-metadata->jsonpickle==1.4.1->-r /tmp/cached-reqsoi878ysp.txt (line 4)) (3.3.0) Requirement already satisfied: retrying>=1.3.3 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from plotly>=3.9.0->trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (1.3.3) Requirement already satisfied: idna<3,>=2.5 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (2.10) Requirement already satisfied: chardet<4,>=3.0.2 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (3.0.4) Collecting orderedmultidict>=1.0.1 Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB) Requirement already satisfied: setuptools in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from jsonschema>=2.6.0->trains==0.16.2->-r /tmp/cached-reqsoi878ysp.txt (line 8)) (49.6.0.post20201009) Processing /home/lv-beast/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl Installing collected packages: jsonpickle, pyserialization, colorama, colorlog, log-utils, tqdm, requests-file, orderedmultidict, furl, pathlib2, humanfriendly, pyrsistent, jsonschema, funcsigs, future, trains, vtk Successfully installed colorama-0.4.3 colorlog-4.4.0 funcsigs-1.0.2 furl-2.1.0 future-0.18.2 humanfriendly-8.2 jsonpickle-1.4.1 jsonschema-3.2.0 log-utils-0.3.4 orderedmultidict-1.0.1 pathlib2-2.3.5 pyrsistent-0.17.3 pyserialization-0.1.1 requests-file-1.5.1 tqdm-4.50.2 trains-0.16.2 vtk-8.1.2 bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Executing Conda: /home/lv-beast/miniconda3/condabin/conda list --json -p /home/lv-beast/.trains/venvs-builds/3.7 Running task id [12dec4fc0c5a44ea8f89e8df293623ef]: [LvObjects/Lv3D/TomoGan/scripts]$ source /home/lv-beast/miniconda3/etc/profile.d/conda.sh && conda activate /home/lv-beast/.trains/venvs-builds/3.7 && /home/lv-beast/.trains/venvs-builds/3.7/bin/python -u train_from_scratch_slices.py Summary - installed python packages: conda:

bmartinn commented 4 years ago

Hi @tomeramit I think the problem is as you mentioned, the astra-toolbox is missing from the "installed packages" You can edit the "Installed packages" and add the missing package (either right-click select reset, or clone the experiment and edit the cloned experiment). The "installed packages" format is equivalent to "requirements.txt" format, meaning that you do not even have to provide a specific version, you can just add a new line with astra-toolbox.

BTW: We recently improved the conda support, an RC should be available soon to test :)

tomeramit commented 4 years ago

I tried to add the astra-toolbox to "Installed packages", it seems a very hacky way (and it didn't work), it's not usable (if I can add the requirements somewhere automatically it's acceptable, but I guess it's not possible)

when you have rc that handle all this conda support (because right now it seems not working...) let me know right now I can't use trains with conda

bmartinn commented 4 years ago

Hi @tomeramit,

Could you test with the latest trains / trains-agent ?

  1. Install the latest trains from the git repo (it improves conda package detection)

    pip install git+https://github.com/allegroai/trains.git

    If you want the full conda enviroment to be logged (indtead of the directly imported packages), in your ~/trains.conf add: sdk.development.detect_with_conda_freeze: true

  2. Re-execute your experiment

  3. Install the latest trains-agent RC (also improving conda support)

    pip install trains-agent==0.16.2rc0

    Make sure that in the trains-agent machine you have conda as package manager: agent.package_manager.type: conda

  4. Enqueue the newly created experiment

tomeramit commented 4 years ago

Hey @bmartinn, thanks for your answer

my trains version is 0.16.2 my trains-agent version is 0.16.2rc0

I'm adding the code of the run (I set sdk.development.detect_with_conda_freeze: true and conda as package manager)

it still doesnt recognize the astra version from some reason, (if it's an issue regarding to the package detection and it will be possible to solve it in this version, I will be really happy :) )

adding logs:

2020-10-18T04:28:14.684Z lv-beast:gpu0,1 INFO task 025c1ec1df0f4195bdfb9396f5fe814d pulled from ca9acd7a32df49e5b4ea41760e252be7 by worker lv-beast:gpu0,1

2020-10-18T04:28:19.717Z lv-beast:gpu0,1 DEBUG Current configuration (trains_agent v0.16.2rc0, location: /tmp/.trains_agent.d4nqoi36.cfg):

sdk.storage.cache.default_base_dir = ~/.trains/cache sdk.storage.cache.size.min_free_bytes = 10GB sdk.storage.direct_access.0.url = file://* sdk.metrics.file_history_size = 100 sdk.metrics.matplotlib_untitled_history_size = 100 sdk.metrics.images.format = JPEG sdk.metrics.images.quality = 87 sdk.metrics.images.subsampling = 0 sdk.metrics.tensorboard_single_series_per_graph = false sdk.network.metrics.file_upload_threads = 4 sdk.network.metrics.file_upload_starvation_warning_sec = 120 sdk.network.iteration.max_retries_on_server_error = 5 sdk.network.iteration.retry_backoff_factor_sec = 10 sdk.aws.s3.key = sdk.aws.s3.region = sdk.aws.boto3.pool_connections = 512 sdk.aws.boto3.max_multipart_concurrency = 16 sdk.log.null_log_propagate = false sdk.log.task_log_buffer_capacity = 66 sdk.log.disable_urllib3_info = true sdk.development.task_reuse_time_window_in_hours = 72.0 sdk.development.vcs_repo_detect_async = true sdk.development.store_uncommitted_code_diff = true sdk.development.support_stopping = true sdk.development.default_output_uri = sdk.development.force_analyze_entire_repo = false sdk.development.suppress_update_message = false sdk.development.detect_with_pip_freeze = false sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true sdk.development.worker.report_global_mem_used = false sdk.development.detect_with_conda_freeze = true agent.worker_id = lv-beast:gpu0,1 agent.worker_name = lv-beast agent.force_git_ssh_protocol = false agent.python_binary = /home/lv-beast/miniconda3/envs/amp/bin/python agent.package_manager.type = conda agent.package_manager.pip_version = <20.2 agent.package_manager.system_site_packages = true agent.package_manager.force_upgrade = false agent.package_manager.conda_channels.0 = simpleitk agent.package_manager.conda_channels.1 = pytorch agent.package_manager.conda_channels.2 = astra-toolbox/label/dev agent.package_manager.conda_channels.3 = conda-forge agent.package_manager.conda_channels.4 = defaults agent.package_manager.torch_nightly = false agent.venvs_dir = /home/lv-beast/.trains/venvs-builds.1 agent.vcs_cache.enabled = true agent.vcs_cache.path = /home/lv-beast/.trains/vcs-cache.1 agent.venv_update.enabled = false agent.pip_download_cache.enabled = true agent.pip_download_cache.path = /home/lv-beast/.trains/pip-download-cache agent.translate_ssh = true agent.reload_config = false agent.docker_pip_cache = /home/lv-beast/.trains/pip-cache agent.docker_apt_cache = /home/lv-beast/.trains/apt-cache.1 agent.docker_force_pull = false agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04 agent.git_user = tomer.amit agent.cuda_version = 101 agent.cudnn_version = 70 agent.default_python = 3.7 api.version = 1.5 api.verify_certificate = true api.default_version = 1.5 api.http.max_req_size = 15728640 api.http.retries.total = 240 api.http.retries.connect = 240 api.http.retries.read = 240 api.http.retries.redirect = 240 api.http.retries.status = 240 api.http.retries.backoff_factor = 1.0 api.http.retries.backoff_max = 120.0 api.http.wait_on_maintenance_forever = true api.http.pool_maxsize = 512 api.http.pool_connections = 512 api.api_server = http://192.168.2.94:8008 api.web_server = http://192.168.2.94:8080 api.files_server = http://192.168.2.94:8081 api.credentials.access_key = IQ4BUOO3Q27ZPICO096M api.host = http://192.168.2.94:8008

Executing task id [025c1ec1df0f4195bdfb9396f5fe814d]: repository = https://bodyvisionmedical.visualstudio.com/DefaultCollection/LungVision/_git/MachineLearning branch = feature/add_amp version_num = 16b77c9b445dcc35b785f731df9d677f6663513c tag = docker_cmd = None entry_point = training_script.py working_dir = LvObjects/Lv3D/TomoGan/scripts

Executing Conda: /home/lv-beast/miniconda3/condabin/conda env remove -p /home/lv-beast/.trains/venvs-builds.1/3.7 --quiet --json

Remove all packages in environment /home/lv-beast/.trains/venvs-builds.1/3.7:

2020-10-18T04:28:44.787Z lv-beast:gpu0,1 DEBUG Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults cudatoolkit=10.1 --quiet --json

2020-10-18T04:30:09.955Z lv-beast:gpu0,1 DEBUG Pass

Using cached repository in "/home/lv-beast/.trains/vcs-cache.1/MachineLearning.6b0aaffa69d428f2574a1d5068d3ee6b/MachineLearning"

2020-10-18T04:30:14.985Z lv-beast:gpu0,1 DEBUG Fetching submodule LungVision Fetching submodule LungVision

2020-10-18T04:30:20.021Z lv-beast:gpu0,1 DEBUG Note: switching to '16b77c9b445dcc35b785f731df9d677f6663513c'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 16b77c9 add envs

2020-10-18T04:30:35.069Z lv-beast:gpu0,1 DEBUG type: git url: https://bodyvisionmedical.visualstudio.com/DefaultCollection/LungVision/_git/MachineLearning branch: HEAD commit: 16b77c9b445dcc35b785f731df9d677f6663513c root: /home/lv-beast/.trains/venvs-builds.1/3.7/task_repository/MachineLearning Applying uncommitted changes

Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults 'pip<20.2' --quiet --json

2020-10-18T04:31:00.129Z lv-beast:gpu0,1 DEBUG Pass Conda: Trying to install requirements: ['certifi~=2020.6.20', 'chardet~=3.0.4', 'cryptography~=3.1.1', 'dask~=2.30.0', 'google_auth~=1.22.0', 'h5py~=2.10.0', 'imgaug~=0.4.0', 'joblib~=0.17.0', 'numba~=0.51.2', 'numpy~=1.16.5', 'packaging~=20.4', 'pandas~=1.1.3', 'Pillow~=7.2.0', 'plotly~=4.11.0', 'psutil~=5.7.2', 'pyclipper~=1.1.0', 'pytorch~=1.6.0', 'requests~=2.24.0', 'scikit_image~=0.16.2', 'scikit_learn~=0.23.2', 'scipy~=1.5.2', 'SimpleITK~=2.0.0', 'tensorboard~=2.3.0', 'torchvision~=0.7.0', 'urllib3~=1.25.10', 'graphviz', 'python-graphviz', 'kiwisolver'] Executing Conda: /home/lv-beast/miniconda3/condabin/conda env update -p /home/lv-beast/.trains/venvs-builds.1/3.7 --file /tmp/conda_env86jwnvvq.yml --quiet --json

2020-10-18T04:31:30.201Z lv-beast:gpu0,1 DEBUG Conda error: ResolvePackageNotFound:

2020-10-18T04:32:30.332Z lv-beast:gpu0,1 DEBUG Pass Conda: Installing requirements: step 2 - using pip: ['git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization', 'colorama==0.4.3', 'colorlog==4.4.0', 'jsonpickle==1.4.1', 'log_utils==0.3.4', 'matplotlib==3.3.2', 'tqdm==4.50.2', 'trains==0.16.2', 'vtk==8.1.2', 'scikit_image~=0.16.2', 'scikit_learn~=0.23.2', 'google_auth~=1.22.0'] Collecting colorama==0.4.3 Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB) Collecting colorlog==4.4.0 Using cached colorlog-4.4.0-py2.py3-none-any.whl (14 kB) Collecting jsonpickle==1.4.1 Using cached jsonpickle-1.4.1-py2.py3-none-any.whl (36 kB) Collecting log_utils==0.3.4 Using cached log_utils-0.3.4-py3-none-any.whl (11 kB) Requirement already satisfied: matplotlib==3.3.2 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (3.3.2) Collecting tqdm==4.50.2 Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB) Collecting trains==0.16.2 Using cached trains-0.16.2-py2.py3-none-any.whl (790 kB) Collecting vtk==8.1.2 Using cached vtk-8.1.2-cp37-cp37m-manylinux1_x86_64.whl (48.9 MB) Collecting scikit_image~=0.16.2 Using cached scikit_image-0.16.2-cp37-cp37m-manylinux1_x86_64.whl (26.5 MB) Collecting scikit_learn~=0.23.2 Using cached scikit_learn-0.23.2-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB) Requirement already satisfied: google_auth~=1.22.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (1.22.0) Requirement already satisfied: importlib-metadata in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from jsonpickle==1.4.1->-r /tmp/cached-reqsbnl_ctg9.txt (line 4)) (2.0.0) Requirement already satisfied: pillow>=6.2.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (7.2.0) Requirement already satisfied: kiwisolver>=1.0.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (1.2.0) Requirement already satisfied: certifi>=2020.06.20 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (2020.6.20) Requirement already satisfied: numpy>=1.15 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (1.16.5) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (2.4.7) Requirement already satisfied: python-dateutil>=2.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (2.8.1) Requirement already satisfied: cycler>=0.10 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 6)) (0.10.0) Processing /home/lv-beast/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl Requirement already satisfied: attrs>=18.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (20.2.0) Requirement already satisfied: PyYAML>=3.12 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (5.3.1) Requirement already satisfied: pyjwt>=1.6.4 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (1.7.1) Collecting requests-file>=1.4.2 Using cached requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB) Collecting jsonschema>=2.6.0 Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB) Requirement already satisfied: urllib3>=1.21.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (1.25.10) Collecting furl>=2.0.0 Using cached furl-2.1.0-py2.py3-none-any.whl (20 kB) Collecting pathlib2>=2.3.0 Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB) Requirement already satisfied: six>=1.11.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (1.15.0) Collecting humanfriendly>=2.1 Using cached humanfriendly-8.2-py2.py3-none-any.whl (86 kB) Requirement already satisfied: requests>=2.20.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (2.24.0) Requirement already satisfied: psutil>=3.4.2 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (5.7.2) Requirement already satisfied: plotly>=3.9.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (4.11.0) Collecting funcsigs>=1.0 Using cached funcsigs-1.0.2-py2.py3-none-any.whl (17 kB) Requirement already satisfied: scipy>=0.19.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 10)) (1.5.2) Requirement already satisfied: networkx>=2.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 10)) (2.5) Requirement already satisfied: imageio>=2.3.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 10)) (2.9.0) Requirement already satisfied: PyWavelets>=0.4.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 10)) (1.1.1) Collecting threadpoolctl>=2.0.0 Using cached threadpoolctl-2.1.0-py3-none-any.whl (12 kB) Requirement already satisfied: joblib>=0.11 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_learn~=0.23.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 11)) (0.17.0) Requirement already satisfied: setuptools>=40.3.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (49.6.0.post20201009) Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (0.2.7) Requirement already satisfied: aiohttp<4.0.0dev,>=3.6.2; python_version >= "3.6" in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (3.6.2) Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.5" in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (4.6) Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (4.1.1) Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from importlib-metadata->jsonpickle==1.4.1->-r /tmp/cached-reqsbnl_ctg9.txt (line 4)) (3.3.1) Processing /home/lv-beast/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl Collecting orderedmultidict>=1.0.1 Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB) Requirement already satisfied: idna<3,>=2.5 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (2.10) Requirement already satisfied: chardet<4,>=3.0.2 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (3.0.4) Requirement already satisfied: retrying>=1.3.3 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from plotly>=3.9.0->trains==0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 8)) (1.3.3) Requirement already satisfied: decorator>=4.3.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from networkx>=2.0->scikit_image~=0.16.2->-r /tmp/cached-reqsbnl_ctg9.txt (line 10)) (4.4.2) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (0.4.8) Requirement already satisfied: yarl<2.0,>=1.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from aiohttp<4.0.0dev,>=3.6.2; python_version >= "3.6"->google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (1.6.2) Requirement already satisfied: async-timeout<4.0,>=3.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from aiohttp<4.0.0dev,>=3.6.2; python_version >= "3.6"->google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (3.0.1) Requirement already satisfied: multidict<5.0,>=4.5 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from aiohttp<4.0.0dev,>=3.6.2; python_version >= "3.6"->google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (4.7.5) Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp<4.0.0dev,>=3.6.2; python_version >= "3.6"->google_auth~=1.22.0->-r /tmp/cached-reqsbnl_ctg9.txt (line 12)) (3.7.4.3) ERROR: imgaug 0.4.0 requires opencv-python-headless, which is not installed. Installing collected packages: colorama, colorlog, jsonpickle, log-utils, tqdm, future, requests-file, pyrsistent, jsonschema, orderedmultidict, furl, pathlib2, humanfriendly, funcsigs, trains, vtk, scikit-image, threadpoolctl, scikit-learn

2020-10-18T04:32:35.360Z lv-beast:gpu0,1 DEBUG Attempting uninstall: scikit-image Found existing installation: scikit-image 0.17.2 Uninstalling scikit-image-0.17.2: Successfully uninstalled scikit-image-0.17.2 Successfully installed colorama-0.4.3 colorlog-4.4.0 funcsigs-1.0.2 furl-2.1.0 future-0.18.2 humanfriendly-8.2 jsonpickle-1.4.1 jsonschema-3.2.0 log-utils-0.3.4 orderedmultidict-1.0.1 pathlib2-2.3.5 pyrsistent-0.17.3 requests-file-1.5.1 scikit-image-0.16.2 scikit-learn-0.23.2 threadpoolctl-2.1.0 tqdm-4.50.2 trains-0.16.2 vtk-8.1.2 Replacing original pip vcs 'git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization' with 'git+https://tomer.amit:xxxxxx@github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization' Processing /home/lv-beast/.cache/pip/wheels/2f/e6/e4/fe5ef74318123672cef559a4cb281c4af196839a116f89c7fd/pyserialization-0.1.1-py3-none-any.whl Requirement already satisfied: numpy in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from pyserialization) (1.16.5) Requirement already satisfied: jsonpickle in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from pyserialization) (1.4.1) Requirement already satisfied: importlib-metadata in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from jsonpickle->pyserialization) (2.0.0) Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from importlib-metadata->jsonpickle->pyserialization) (3.3.1) Installing collected packages: pyserialization Successfully installed pyserialization-0.1.1 Executing Conda: /home/lv-beast/miniconda3/condabin/conda list --json -p /home/lv-beast/.trains/venvs-builds.1/3.7 Running task id [025c1ec1df0f4195bdfb9396f5fe814d]: [LvObjects/Lv3D/TomoGan/scripts]$ source /home/lv-beast/miniconda3/etc/profile.d/conda.sh && conda activate /home/lv-beast/.trains/venvs-builds.1/3.7 && /home/lv-beast/.trains/venvs-builds.1/3.7/bin/python -u training_script.py Summary - installed python packages: conda:

Environment setup completed successfully

Starting Task Execution:

2020-10-18T04:32:40.393Z lv-beast:gpu0,1 DEBUG Traceback (most recent call last): File "training_script.py", line 23, in from tomo.projectors.parallel_projection import ParallelProjection File "/home/lv-beast/.trains/venvs-builds.1/3.7/task_repository/MachineLearning/LungVision/tomo/tomo/projectors/parallel_projection.py", line 1, in import astra ModuleNotFoundError: No module named 'astra'

tomeramit commented 4 years ago

from astra page: https://www.astra-toolbox.com/downloads/index.html conda install -c astra-toolbox/label/dev astra-toolbox (I added the channel as you can see in the logs) this is my installed version: astra-toolbox 1.9.0.dev11 np116py37_0 astra-toolbox/label/dev

bmartinn commented 4 years ago

Hi @tomeramit Just making sure, this test was done after you uninstalled the trains package on the development machine, then reinstalled trains from the git repository (i.e. "pip install git+https://github.com/allegroai/trains.git"), then rerun the code? Could you also provide the "installed packages" before you cloned the experiment and enqueued it for execution by the trains-agent ?

To summarize:

tomeramit commented 4 years ago

Hi @bmartinn

Pillow == 7.2.0 SimpleITK == 2.0.0 certifi == 2020.6.20 chardet == 3.0.4 colorama == 0.4.3 colorlog == 4.4.0 cryptography == 3.1.1 dask == 2.30.0 h5py == 2.10.0 imgaug == 0.4.0 joblib == 0.17.0 jsonpickle == 1.4.1 log_utils == 0.3.4 matplotlib == 3.3.2 numba == 0.51.2 numpy == 1.16.5 pandas == 1.1.3 pip == 20.2.3 plotly == 4.11.0 protobuf == 3.13.0 psutil == 5.7.2 pyclipper == 1.1.0.post1 pyserialization @ git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization requests == 2.24.0 scikit_image == 0.17.2 scikit_learn == 0.23.2 scipy == 1.5.2 tensorboard == 2.3.0 torch == 1.6.0 torchvision == 0.7.0 tqdm == 4.50.2 trains @ git+https://github.com/allegroai/trains.git@4f303cb37b3deb082a519043791847a48424861d#egg=trains urllib3 == 1.25.10 vtk == 8.1.2

the script of launching the trains agent is: ###########################

!/bin/sh

. /home/lv-beast/miniconda3/etc/profile.d/conda.sh conda activate trains-agent trains-agent daemon --gpus 0,1 --queue dual_gpu --foreground ############################ (I assigned the experiment to the dual_gpu queue)

logs: 2020-10-19T08:12:15.925Z lv-beast:gpu0,1 INFO task 75e61e926c1144d1961e2442181ff714 pulled from ca9acd7a32df49e5b4ea41760e252be7 by worker lv-beast:gpu0,1

2020-10-19T08:12:20.960Z lv-beast:gpu0,1 DEBUG Current configuration (trains_agent v0.16.2rc0, location: /tmp/.trains_agent.6uqmi9_b.cfg):

sdk.storage.cache.default_base_dir = ~/.trains/cache sdk.storage.cache.size.min_free_bytes = 10GB sdk.storage.direct_access.0.url = file://* sdk.metrics.file_history_size = 100 sdk.metrics.matplotlib_untitled_history_size = 100 sdk.metrics.images.format = JPEG sdk.metrics.images.quality = 87 sdk.metrics.images.subsampling = 0 sdk.metrics.tensorboard_single_series_per_graph = false sdk.network.metrics.file_upload_threads = 4 sdk.network.metrics.file_upload_starvation_warning_sec = 120 sdk.network.iteration.max_retries_on_server_error = 5 sdk.network.iteration.retry_backoff_factor_sec = 10 sdk.aws.s3.key = sdk.aws.s3.region = sdk.aws.boto3.pool_connections = 512 sdk.aws.boto3.max_multipart_concurrency = 16 sdk.log.null_log_propagate = false sdk.log.task_log_buffer_capacity = 66 sdk.log.disable_urllib3_info = true sdk.development.task_reuse_time_window_in_hours = 72.0 sdk.development.vcs_repo_detect_async = true sdk.development.store_uncommitted_code_diff = true sdk.development.support_stopping = true sdk.development.default_output_uri = sdk.development.force_analyze_entire_repo = false sdk.development.suppress_update_message = false sdk.development.detect_with_pip_freeze = false sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true sdk.development.worker.report_global_mem_used = false sdk.development.detect_with_conda_freeze = true agent.worker_id = lv-beast:gpu0,1 agent.worker_name = lv-beast agent.force_git_ssh_protocol = false agent.python_binary = /home/lv-beast/miniconda3/envs/amp/bin/python agent.package_manager.type = conda agent.package_manager.pip_version = <20.2 agent.package_manager.system_site_packages = true agent.package_manager.force_upgrade = false agent.package_manager.conda_channels.0 = simpleitk agent.package_manager.conda_channels.1 = pytorch agent.package_manager.conda_channels.2 = astra-toolbox/label/dev agent.package_manager.conda_channels.3 = conda-forge agent.package_manager.conda_channels.4 = defaults agent.package_manager.torch_nightly = false agent.venvs_dir = /home/lv-beast/.trains/venvs-builds agent.vcs_cache.enabled = true agent.vcs_cache.path = /home/lv-beast/.trains/vcs-cache agent.venv_update.enabled = false agent.pip_download_cache.enabled = true agent.pip_download_cache.path = /home/lv-beast/.trains/pip-download-cache agent.translate_ssh = true agent.reload_config = false agent.docker_pip_cache = /home/lv-beast/.trains/pip-cache agent.docker_apt_cache = /home/lv-beast/.trains/apt-cache agent.docker_force_pull = false agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04 agent.git_user = tomer.amit agent.cuda_version = 101 agent.cudnn_version = 70 agent.default_python = 3.7 api.version = 1.5 api.verify_certificate = true api.default_version = 1.5 api.http.max_req_size = 15728640 api.http.retries.total = 240 api.http.retries.connect = 240 api.http.retries.read = 240 api.http.retries.redirect = 240 api.http.retries.status = 240 api.http.retries.backoff_factor = 1.0 api.http.retries.backoff_max = 120.0 api.http.wait_on_maintenance_forever = true api.http.pool_maxsize = 512 api.http.pool_connections = 512 api.api_server = http://192.168.2.94:8008 api.web_server = http://192.168.2.94:8080 api.files_server = http://192.168.2.94:8081 api.credentials.access_key = IQ4BUOO3Q27ZPICO096M api.host = http://192.168.2.94:8008

Executing task id [75e61e926c1144d1961e2442181ff714]: repository = https://bodyvisionmedical.visualstudio.com/LungVision/Streamline/_git/MachineLearning branch = feature/add_amp version_num = 16b77c9b445dcc35b785f731df9d677f6663513c tag = docker_cmd = None entry_point = train_from_scratch_slices.py working_dir = LvObjects/Lv3D/TomoGan/scripts

Executing Conda: /home/lv-beast/miniconda3/condabin/conda env remove -p /home/lv-beast/.trains/venvs-builds/3.7 --quiet --json

2020-10-19T08:12:51.025Z lv-beast:gpu0,1 DEBUG Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults cudatoolkit=10.1 --quiet --json

2020-10-19T08:13:16.093Z lv-beast:gpu0,1 DEBUG Pass

Using cached repository in "/home/lv-beast/.trains/vcs-cache/MachineLearning.19ae66b07920a67ff810b2ac379f9292/MachineLearning" From https://bodyvisionmedical.visualstudio.com/LungVision/Streamline/_git/MachineLearning 80f10a3..2c59ec6 develop -> origin/develop

2020-10-19T08:13:21.119Z lv-beast:gpu0,1 DEBUG From https://bodyvisionmedical.visualstudio.com/DefaultCollection/_git/LungVision

2020-10-19T08:13:26.144Z lv-beast:gpu0,1 DEBUG Note: switching to '16b77c9b445dcc35b785f731df9d677f6663513c'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 16b77c9 add envs

2020-10-19T08:13:41.184Z lv-beast:gpu0,1 DEBUG type: git url: https://bodyvisionmedical.visualstudio.com/LungVision/Streamline/_git/MachineLearning branch: HEAD commit: 16b77c9b445dcc35b785f731df9d677f6663513c root: /home/lv-beast/.trains/venvs-builds/3.7/task_repository/MachineLearning

Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults 'pip<20.2' --quiet --json

2020-10-19T08:14:06.241Z lv-beast:gpu0,1 DEBUG Pass Conda: Trying to install requirements: ['certifi~=2020.6.20', 'chardet~=3.0.4', 'cryptography~=3.1.1', 'h5py~=2.10.0', 'imgaug~=0.4.0', 'joblib~=0.17.0', 'numba~=0.51.2', 'numpy~=1.16.5', 'pandas~=1.1.3', 'Pillow~=7.2.0', 'plotly~=4.11.0', 'protobuf~=3.13.0', 'pyclipper~=1.1.0', 'pytorch~=1.6.0', 'requests~=2.24.0', 'scikit_image~=0.17.2', 'scikit_learn~=0.23.2', 'scipy~=1.5.2', 'SimpleITK~=2.0.0', 'tensorboard~=2.3.0', 'torchvision~=0.7.0', 'urllib3~=1.25.10', 'graphviz', 'python-graphviz', 'kiwisolver'] Executing Conda: /home/lv-beast/miniconda3/condabin/conda env update -p /home/lv-beast/.trains/venvs-builds/3.7 --file /tmp/conda_envij3ikpoz.yml --quiet --json

2020-10-19T08:14:51.336Z lv-beast:gpu0,1 DEBUG Conda error: ResolvePackageNotFound:

2020-10-19T08:15:51.457Z lv-beast:gpu0,1 DEBUG Pass Conda: Installing requirements: step 2 - using pip: ['colorama==0.4.3', 'colorlog==4.4.0', 'dask==2.30.0', 'jsonpickle==1.4.1', 'log_utils==0.3.4', 'matplotlib==3.3.2', 'psutil==5.7.2', 'git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization', 'tqdm==4.50.2', 'git+https://github.com/allegroai/trains.git@4f303cb37b3deb082a519043791847a48424861d#egg=trains', 'vtk==8.1.2', 'scikit_image~=0.17.2', 'scikit_learn~=0.23.2'] bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Collecting colorama==0.4.3 Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB) Collecting colorlog==4.4.0 Using cached colorlog-4.4.0-py2.py3-none-any.whl (14 kB) Requirement already satisfied: dask==2.30.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqsursnd7m3.txt (line 3)) (2.30.0) Collecting jsonpickle==1.4.1 Using cached jsonpickle-1.4.1-py2.py3-none-any.whl (36 kB) Collecting log_utils==0.3.4 Using cached log_utils-0.3.4-py3-none-any.whl (11 kB) Requirement already satisfied: matplotlib==3.3.2 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqsursnd7m3.txt (line 6)) (3.3.2) Processing /home/lv-beast/.cache/pip/wheels/2d/43/97/00701864a7bee6d9e1a52dd682537dcbf1d013d0e2e6f0c1f1/psutil-5.7.2-cp37-cp37m-linux_x86_64.whl Collecting tqdm==4.50.2 Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB) Collecting vtk==8.1.2 Using cached vtk-8.1.2-cp37-cp37m-manylinux1_x86_64.whl (48.9 MB) Requirement already satisfied: scikit_image~=0.17.2 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqsursnd7m3.txt (line 12)) (0.17.2)

2020-10-19T08:15:56.476Z lv-beast:gpu0,1 DEBUG Collecting scikit_learn~=0.23.2 Using cached scikit_learn-0.23.2-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB) Requirement already satisfied: pyyaml in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from dask==2.30.0->-r /tmp/cached-reqsursnd7m3.txt (line 3)) (5.3.1) Requirement already satisfied: importlib-metadata in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from jsonpickle==1.4.1->-r /tmp/cached-reqsursnd7m3.txt (line 4)) (2.0.0) Requirement already satisfied: certifi>=2020.06.20 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (2020.6.20) Requirement already satisfied: pillow>=6.2.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (7.2.0) Requirement already satisfied: cycler>=0.10 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (0.10.0) Requirement already satisfied: python-dateutil>=2.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (2.8.1) Requirement already satisfied: numpy>=1.15 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (1.16.5) Requirement already satisfied: kiwisolver>=1.0.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (1.2.0) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (2.4.7) Requirement already satisfied: scipy>=1.0.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqsursnd7m3.txt (line 12)) (1.5.2) Requirement already satisfied: networkx>=2.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqsursnd7m3.txt (line 12)) (2.5) Requirement already satisfied: imageio>=2.3.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqsursnd7m3.txt (line 12)) (2.9.0) Requirement already satisfied: tifffile>=2019.7.26 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqsursnd7m3.txt (line 12)) (2020.10.1) Requirement already satisfied: PyWavelets>=1.1.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqsursnd7m3.txt (line 12)) (1.1.1) Requirement already satisfied: joblib>=0.11 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from scikit_learn~=0.23.2->-r /tmp/cached-reqsursnd7m3.txt (line 13)) (0.17.0) Collecting threadpoolctl>=2.0.0 Using cached threadpoolctl-2.1.0-py3-none-any.whl (12 kB) Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from importlib-metadata->jsonpickle==1.4.1->-r /tmp/cached-reqsursnd7m3.txt (line 4)) (3.3.1) Requirement already satisfied: six in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from cycler>=0.10->matplotlib==3.3.2->-r /tmp/cached-reqsursnd7m3.txt (line 6)) (1.15.0) Requirement already satisfied: decorator>=4.3.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from networkx>=2.0->scikit_image~=0.17.2->-r /tmp/cached-reqsursnd7m3.txt (line 12)) (4.4.2) Installing collected packages: colorama, colorlog, jsonpickle, log-utils, psutil, tqdm, vtk, threadpoolctl, scikit-learn Successfully installed colorama-0.4.3 colorlog-4.4.0 jsonpickle-1.4.1 log-utils-0.3.4 psutil-5.7.2 scikit-learn-0.23.2 threadpoolctl-2.1.0 tqdm-4.50.2 vtk-8.1.2 bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Replacing original pip vcs 'git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization' with 'git+https://tomer.amit:xxxxxx@github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization' bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Processing /home/lv-beast/.cache/pip/wheels/2f/e6/e4/fe5ef74318123672cef559a4cb281c4af196839a116f89c7fd/pyserialization-0.1.1-py3-none-any.whl Requirement already satisfied: jsonpickle in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from pyserialization) (1.4.1) Requirement already satisfied: numpy in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from pyserialization) (1.16.5) Requirement already satisfied: importlib-metadata in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from jsonpickle->pyserialization) (2.0.0) Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from importlib-metadata->jsonpickle->pyserialization) (3.3.1) Installing collected packages: pyserialization Successfully installed pyserialization-0.1.1 bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Replacing original pip vcs 'git+https://github.com/allegroai/trains.git@4f303cb37b3deb082a519043791847a48424861d#egg=trains' with 'git+https://tomer.amit:xxxxxx@github.com/allegroai/trains.git@4f303cb37b3deb082a519043791847a48424861d#egg=trains' bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Collecting trains Cloning https://tomer.amit:****@github.com/allegroai/trains.git (to revision 4f303cb37b3deb082a519043791847a48424861d) to /tmp/pip-install-q64wp30o/trains Running command git clone -q 'https://tomer.amit:****@github.com/allegroai/trains.git' /tmp/pip-install-q64wp30o/trains

2020-10-19T08:16:01.496Z lv-beast:gpu0,1 DEBUG Requirement already satisfied: attrs>=18.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (20.2.0) Collecting funcsigs>=1.0 Using cached funcsigs-1.0.2-py2.py3-none-any.whl (17 kB) Collecting furl>=2.0.0 Using cached furl-2.1.0-py2.py3-none-any.whl (20 kB) Processing /home/lv-beast/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl Collecting humanfriendly>=2.1 Using cached humanfriendly-8.2-py2.py3-none-any.whl (86 kB) Collecting jsonschema>=2.6.0 Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB) Requirement already satisfied: numpy>=1.10 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (1.16.5) Collecting pathlib2>=2.3.0 Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB) Requirement already satisfied: Pillow>=4.1.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (7.2.0) Requirement already satisfied: plotly>=3.9.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (4.11.0) Requirement already satisfied: psutil>=3.4.2 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (5.7.2) Requirement already satisfied: pyparsing>=2.0.3 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (2.4.7) Requirement already satisfied: python-dateutil>=2.6.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (2.8.1) Requirement already satisfied: pyjwt>=1.6.4 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (1.7.1) Requirement already satisfied: PyYAML>=3.12 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (5.3.1) Collecting requests-file>=1.4.2 Using cached requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB) Requirement already satisfied: requests>=2.20.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (2.24.0) Requirement already satisfied: six>=1.11.0 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (1.15.0) Requirement already satisfied: tqdm>=4.19.5 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (4.50.2) Requirement already satisfied: urllib3>=1.21.1 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from trains) (1.25.10) Collecting orderedmultidict>=1.0.1 Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB) Processing /home/lv-beast/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl Requirement already satisfied: setuptools in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from jsonschema>=2.6.0->trains) (49.6.0.post20201009) Requirement already satisfied: importlib-metadata; python_version < "3.8" in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from jsonschema>=2.6.0->trains) (2.0.0) Requirement already satisfied: retrying>=1.3.3 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from plotly>=3.9.0->trains) (1.3.3) Requirement already satisfied: certifi>=2017.4.17 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains) (2020.6.20) Requirement already satisfied: idna<3,>=2.5 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains) (2.10) Requirement already satisfied: chardet<4,>=3.0.2 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains) (3.0.4) Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds/3.7/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema>=2.6.0->trains) (3.3.1) Building wheels for collected packages: trains Building wheel for trains (setup.py): started Building wheel for trains (setup.py): finished with status 'done' Created wheel for trains: filename=trains-0.16.2-py2.py3-none-any.whl size=801299 sha256=3fa25ddc07ef9fff388c68de2f1cde99eb37335b51ece42687e9aa35ebe98784 Stored in directory: /home/lv-beast/.cache/pip/wheels/90/0e/8b/e428631fd3e335245fefceba0281992b68db3098de4624bfe5 Successfully built trains Installing collected packages: funcsigs, orderedmultidict, furl, future, humanfriendly, pyrsistent, jsonschema, pathlib2, requests-file, trains Successfully installed funcsigs-1.0.2 furl-2.1.0 future-0.18.2 humanfriendly-8.2 jsonschema-3.2.0 orderedmultidict-1.0.1 pathlib2-2.3.5 pyrsistent-0.17.3 requests-file-1.5.1 trains-0.16.2 bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Executing Conda: /home/lv-beast/miniconda3/condabin/conda list --json -p /home/lv-beast/.trains/venvs-builds/3.7

2020-10-19T08:16:06.521Z lv-beast:gpu0,1 DEBUG Running task id [75e61e926c1144d1961e2442181ff714]: [LvObjects/Lv3D/TomoGan/scripts]$ source /home/lv-beast/miniconda3/etc/profile.d/conda.sh && conda activate /home/lv-beast/.trains/venvs-builds/3.7 && /home/lv-beast/.trains/venvs-builds/3.7/bin/python -u train_from_scratch_slices.py Summary - installed python packages: conda:

Environment setup completed successfully

bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Starting Task Execution:

bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Traceback (most recent call last): File "train_from_scratch_slices.py", line 21, in from LvObjects.Lv3D.TomoGan.scripts.training_script import create_hyper_params_cyclegan_from_path, create_tomogan_flow File "/home/lv-beast/.trains/venvs-builds/3.7/task_repository/MachineLearning/LvObjects/Lv3D/TomoGan/scripts/training_script.py", line 23, in from tomo.projectors.parallel_projection import ParallelProjection File "/home/lv-beast/.trains/venvs-builds/3.7/task_repository/MachineLearning/LungVision/tomo/tomo/projectors/parallel_projection.py", line 1, in import astra ModuleNotFoundError: No module named 'astra'

bmartinn commented 4 years ago

Hi @tomeramit I think the main issue is the "astra-toolbox" does not appear in the original "installed packages". When running with sdk.development.detect_with_conda_freeze: true set in your ~/trains.conf, The installed packages should look something like:

# Python 3.6.6 |Anaconda, Inc.| (default, Oct  9 2018, 12:34:16)  [GCC 7.3.0]

aadict==0.2.3
<more pypi packages here>...

# Conda Packages

absl-py==0.10.0
astor==0.8.1
astra-toolbox==1.9.9.dev4

But I could not locate the "# Conda Packages" in your "installed packages" section.

I think the issue is the pip install (it never reinstalled, it used the cached wheel). Could you retry again, this time we add the -U flag making it reinstall from the git, and not use the cached wheel:

tomeramit commented 4 years ago

I'm not sure what you meant, but I did reinstall trains on my develop machine (version is still 0.16.2) if there is more advanced version, let me know re run the code and same thing happened (Adding again the installed packages

if you meant something else and I misunderstood let me know

thanks

Python 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 01:53:57) [MSC v.1916 64 bit (AMD64)]

Pillow == 7.2.0 SimpleITK == 2.0.0 certifi == 2020.6.20 chardet == 3.0.4 colorama == 0.4.3 colorlog == 4.4.0 cryptography == 3.1.1 dask == 2.30.0 h5py == 2.10.0 imgaug == 0.4.0 joblib == 0.17.0 jsonpickle == 1.4.1 log_utils == 0.3.4 matplotlib == 3.3.2 numba == 0.51.2 numpy == 1.16.5 pandas == 1.1.3 pip == 20.2.3 plotly == 4.11.0 protobuf == 3.13.0 psutil == 5.7.2 pyclipper == 1.1.0.post1 pyserialization @ git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization requests == 2.24.0 scikit_image == 0.17.2 scikit_learn == 0.23.2 scipy == 1.5.2 tensorboard == 2.3.0 torch == 1.6.0 torchvision == 0.7.0 tqdm == 4.50.2 trains @ git+https://github.com/allegroai/trains.git@04b3fa809bb73d7101d1995327684ebe5b2911e3#egg=trains urllib3 == 1.25.10 vtk == 8.1.2

Detailed import analysis

**

bmartinn commented 4 years ago

@tomeramit this looks okay, trains was definitely installed from the correct git repo , and the commit is the latest one ... It seems to totally ignore the sdk.development.detect_with_conda_freeze: true in trains.conf

Are you positive you have it in the development machine's trains.conf ? (From the log I see you have it on the remote machine, running trains-agent. Actually it has not effect on trains-agent, it is only important for trains while running the code on the development machine)

If this is not the case, the only thing I can think of is maybe something to do with running on Windows ?!

What do you think?

tomeramit commented 4 years ago

oh... I did it only on the remote machine I will try to change it also on the development machine there are other things that I have to set on the development machine? (because I didn't change it at all)

bmartinn commented 4 years ago

oh... I did it only on the remote machine

No worries, I probably should have stressed it better :)

Once you add the sdk.development.detect_with_conda_freeze: true to the development machine's trains.conf (and of course make sure trains is installed from the git repo). Then just rerun the code, and check if in the newly generated "installed packages" you see # Conda Packages (no need to send for remote execution, we are just making sure the "dev" machine extracted all the conda packages) :crossed_fingers:

tomeramit commented 4 years ago

so first of all it worked! (when I launched from linux computer) when I try to launch from my dev (windows) it has conflicts (because of some windows lib that arent necassery for linux at all (like visualstudio things, that have another replacement for linux) it's possible to overcome this issue? (adding logs)

2020-10-22T04:35:42.911Z lv-beast:gpu0,1 INFO task 51a5211276604bbdab78a08b78440856 pulled from ca9acd7a32df49e5b4ea41760e252be7 by worker lv-beast:gpu0,1

2020-10-22T04:35:47.945Z lv-beast:gpu0,1 DEBUG Current configuration (trains_agent v0.16.2rc0, location: /tmp/.trains_agent.y6x360l5.cfg):

sdk.storage.cache.default_base_dir = ~/.trains/cache sdk.storage.cache.size.min_free_bytes = 10GB sdk.storage.direct_access.0.url = file://* sdk.metrics.file_history_size = 100 sdk.metrics.matplotlib_untitled_history_size = 100 sdk.metrics.images.format = JPEG sdk.metrics.images.quality = 87 sdk.metrics.images.subsampling = 0 sdk.metrics.tensorboard_single_series_per_graph = false sdk.network.metrics.file_upload_threads = 4 sdk.network.metrics.file_upload_starvation_warning_sec = 120 sdk.network.iteration.max_retries_on_server_error = 5 sdk.network.iteration.retry_backoff_factor_sec = 10 sdk.aws.s3.key = sdk.aws.s3.region = sdk.aws.boto3.pool_connections = 512 sdk.aws.boto3.max_multipart_concurrency = 16 sdk.log.null_log_propagate = false sdk.log.task_log_buffer_capacity = 66 sdk.log.disable_urllib3_info = true sdk.development.task_reuse_time_window_in_hours = 72.0 sdk.development.vcs_repo_detect_async = true sdk.development.store_uncommitted_code_diff = true sdk.development.support_stopping = true sdk.development.default_output_uri = sdk.development.force_analyze_entire_repo = false sdk.development.suppress_update_message = false sdk.development.detect_with_pip_freeze = false sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true sdk.development.worker.report_global_mem_used = false sdk.development.detect_with_conda_freeze = true agent.worker_id = lv-beast:gpu0,1 agent.worker_name = lv-beast agent.force_git_ssh_protocol = false agent.python_binary = /home/lv-beast/miniconda3/envs/keras-dl/bin/python agent.package_manager.type = conda agent.package_manager.pip_version = <20.2 agent.package_manager.system_site_packages = true agent.package_manager.force_upgrade = false agent.package_manager.conda_channels.0 = simpleitk agent.package_manager.conda_channels.1 = pytorch agent.package_manager.conda_channels.2 = astra-toolbox/label/dev agent.package_manager.conda_channels.3 = conda-forge agent.package_manager.conda_channels.4 = defaults agent.package_manager.torch_nightly = false agent.venvs_dir = /home/lv-beast/.trains/venvs-builds.1 agent.vcs_cache.enabled = true agent.vcs_cache.path = /home/lv-beast/.trains/vcs-cache.1 agent.venv_update.enabled = false agent.pip_download_cache.enabled = true agent.pip_download_cache.path = /home/lv-beast/.trains/pip-download-cache agent.translate_ssh = true agent.reload_config = false agent.docker_pip_cache = /home/lv-beast/.trains/pip-cache agent.docker_apt_cache = /home/lv-beast/.trains/apt-cache.1 agent.docker_force_pull = false agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04 agent.git_user = tomer.amit agent.cuda_version = 101 agent.cudnn_version = 70 agent.default_python = 3.7 api.version = 1.5 api.verify_certificate = true api.default_version = 1.5 api.http.max_req_size = 15728640 api.http.retries.total = 240 api.http.retries.connect = 240 api.http.retries.read = 240 api.http.retries.redirect = 240 api.http.retries.status = 240 api.http.retries.backoff_factor = 1.0 api.http.retries.backoff_max = 120.0 api.http.wait_on_maintenance_forever = true api.http.pool_maxsize = 512 api.http.pool_connections = 512 api.api_server = http://192.168.2.94:8008 api.web_server = http://192.168.2.94:8080 api.files_server = http://192.168.2.94:8081 api.credentials.access_key = IQ4BUOO3Q27ZPICO096M api.host = http://192.168.2.94:8008

Executing task id [51a5211276604bbdab78a08b78440856]: repository = https://bodyvisionmedical.visualstudio.com/LungVision/_git/MachineLearning branch = feature/add_amp version_num = 16b77c9b445dcc35b785f731df9d677f6663513c tag = docker_cmd = entry_point = training_script.py working_dir = LvObjects/Lv3D/TomoGan/scripts

Executing Conda: /home/lv-beast/miniconda3/condabin/conda env remove -p /home/lv-beast/.trains/venvs-builds.1/3.7 --quiet --json

Remove all packages in environment /home/lv-beast/.trains/venvs-builds.1/3.7:

2020-10-22T04:36:18.005Z lv-beast:gpu0,1 DEBUG Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults cudatoolkit=10.1 --quiet --json

2020-10-22T04:36:43.076Z lv-beast:gpu0,1 DEBUG Pass

Using cached repository in "/home/lv-beast/.trains/vcs-cache.1/MachineLearning.2f27b16d384a01d04c7eedd7180ff087/MachineLearning"

2020-10-22T04:36:48.100Z lv-beast:gpu0,1 DEBUG Fetching submodule LungVision Fetching submodule LungVision Note: switching to '16b77c9b445dcc35b785f731df9d677f6663513c'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 16b77c9 add envs

2020-10-22T04:37:03.165Z lv-beast:gpu0,1 DEBUG type: git url: https://bodyvisionmedical.visualstudio.com/LungVision/_git/MachineLearning branch: HEAD commit: 16b77c9b445dcc35b785f731df9d677f6663513c root: /home/lv-beast/.trains/venvs-builds.1/3.7/task_repository/MachineLearning

Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults 'pip<20.2' --quiet --json

2020-10-22T04:37:33.233Z lv-beast:gpu0,1 DEBUG Pass Conda: Trying to install requirements: ['absl-py~=0.10.0', 'aiohttp~=3.6.2', 'astra-toolbox~=1.9.0.dev11', 'async-timeout~=3.0.1', 'attrs~=20.2.0', 'blas~=1.0', 'blinker~=1.4', 'blosc~=1.20.1', 'brotli~=1.0.9', 'brotlipy~=0.7.0', 'bzip2~=1.0.8', 'ca-certificates~=2020.6.20', 'cachetools~=4.1.1', 'certifi~=2020.6.20', 'cffi~=1.14.3', 'chardet~=3.0.4', 'charls~=2.1.0', 'click~=7.1.2', 'cloudpickle~=1.6.0', 'cryptography~=3.1.1', 'cycler~=0.10.0', 'cytoolz~=0.11.0', 'dask-core~=2.30.0', 'decorator~=4.4.2', 'freetype~=2.10.3', 'geos~=3.8.1', 'giflib~=5.2.1', 'google-auth~=1.22.0', 'google-auth-oauthlib~=0.4.1', 'grpcio~=1.31.0', 'h5py~=2.10.0', 'hdf5~=1.10.6', 'icc_rt~=2019.0.0', 'icu~=67.1', 'idna~=2.10', 'imagecodecs~=2020.5.30', 'imageio~=2.9.0', 'imgaug~=0.4.0', 'importlib-metadata~=2.0.0', 'intel-openmp~=2019.4', 'joblib~=0.17.0', 'jpeg~=9d.0', 'jxrlib~=1.1', 'kiwisolver~=1.2.0', 'lcms2~=2.11', 'lerc~=2.2', 'libaec~=1.0.4', 'libastra~=1.9.0.dev11', 'libblas~=3.8.0', 'libcblas~=3.8.0', 'libclang~=10.0.1', 'liblapack~=3.8.0', 'liblapacke~=3.8.0', 'libopencv~=4.4.0', 'libpng~=1.6.37', 'libprotobuf~=3.13.0.1', 'libtiff~=4.1.0', 'libwebp-base~=1.1.0', 'libzopfli~=1.0.3', 'llvmlite~=0.34.0', 'lz4-c~=1.9.2', 'markdown~=3.3', 'matplotlib-base~=3.3.2', 'mkl~=2019.4', 'mkl-service~=2.3.0', 'multidict~=4.7.5', 'networkx~=2.5', 'ninja~=1.10.1', 'numba~=0.51.2', 'numpy~=1.16.5', 'oauthlib~=3.0.1', 'olefile~=0.46', 'opencv~=4.4.0', 'openjpeg~=2.3.1', 'openssl~=1.1.1h', 'pandas~=1.1.3', 'pillow~=7.2.0', 'plotly~=4.11.0', 'protobuf~=3.13.0.1', 'py-opencv~=4.4.0', 'pyasn1~=0.4.8', 'pyasn1-modules~=0.2.7', 'pyclipper~=1.1.0', 'pycparser~=2.20', 'pyjwt~=1.7.1', 'pyopenssl~=19.1.0', 'pyparsing~=2.4.7', 'pyreadline~=2.1', 'pysocks~=1.7.1', 'python~=3.7.8', 'python-dateutil~=2.8.1', 'python_abi~=3.7', 'pytorch~=1.6.0', 'pytz~=2020.1', 'pywavelets~=1.1.1', 'pyyaml~=5.3.1', 'qt~=5.12.9', 'requests~=2.24.0', 'requests-oauthlib~=1.3.0', 'retrying~=1.3.3', 'rsa~=4.6', 'scikit-image~=0.17.2', 'scikit-learn~=0.23.2', 'scipy~=1.5.2', 'setuptools~=49.6.0', 'shapely~=1.7.1', 'simpleitk~=2.0.0', 'six~=1.15.0', 'snappy~=1.1.8', 'sqlite~=3.33.0', 'tensorboard~=2.3.0', 'tensorboard-plugin-wit~=1.6.0', 'threadpoolctl~=2.1.0', 'tifffile~=2020.10.1', 'tk~=8.6.10', 'toolz~=0.11.1', 'torchvision~=0.7.0', 'tornado~=6.0.4', 'urllib3~=1.25.10', 'vc~=14.1', 'vs2015_runtime~=14.16.27012', 'werkzeug~=1.0.1', 'wheel~=0.35.1', 'win_inet_pton~=1.1.0', 'wincertstore~=0.2', 'xz~=5.2.5', 'yaml~=0.2.5', 'yarl~=1.3.0', 'zfp~=0.5.5', 'zipp~=3.3.0', 'zlib~=1.2.11', 'zstd~=1.4.5'] Executing Conda: /home/lv-beast/miniconda3/condabin/conda env update -p /home/lv-beast/.trains/venvs-builds.1/3.7 --file /tmp/conda_envazadbunc.yml --quiet --json

2020-10-22T04:38:13.317Z lv-beast:gpu0,1 DEBUG Conda error: ResolvePackageNotFound:

2020-10-22T04:39:38.497Z lv-beast:gpu0,1 DEBUG Pass Conda: Installing requirements: step 2 - using pip: ['colorama==0.4.3', 'colorlog==4.4.0', 'funcsigs==1.0.2', 'furl==2.1.0', 'future==0.18.2', 'humanfriendly==8.2', 'jsonpickle==1.4.1', 'jsonschema==3.2.0', 'log-utils==0.3.4', 'orderedmultidict==1.0.1', 'pathlib2==2.3.5', 'psutil==5.7.2', 'pyrsistent==0.17.3', 'git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad', 'requests-file==1.5.1', 'tqdm==4.50.2', 'git+https://github.com/allegroai/trains.git@04b3fa809bb73d7101d1995327684ebe5b2911e3', 'vtk==8.1.2', 'icc_rt~=2019.0.0', 'vs2015_runtime~=14.16.27012', 'wincertstore~=0.2', 'vc~=14.1', 'pyreadline~=2.1', 'win_inet_pton~=1.1.0'] bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Collecting colorama==0.4.3 Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB)

2020-10-22T04:39:43.517Z lv-beast:gpu0,1 DEBUG Collecting colorlog==4.4.0 Using cached colorlog-4.4.0-py2.py3-none-any.whl (14 kB) Collecting funcsigs==1.0.2 Using cached funcsigs-1.0.2-py2.py3-none-any.whl (17 kB) Collecting furl==2.1.0 Using cached furl-2.1.0-py2.py3-none-any.whl (20 kB) Processing /home/lv-beast/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl Collecting humanfriendly==8.2 Using cached humanfriendly-8.2-py2.py3-none-any.whl (86 kB) Collecting jsonpickle==1.4.1 Using cached jsonpickle-1.4.1-py2.py3-none-any.whl (36 kB) Collecting jsonschema==3.2.0 Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB) Collecting log-utils==0.3.4 Using cached log_utils-0.3.4-py3-none-any.whl (11 kB) Collecting orderedmultidict==1.0.1 Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB) Collecting pathlib2==2.3.5 Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB) Processing /home/lv-beast/.cache/pip/wheels/2d/43/97/00701864a7bee6d9e1a52dd682537dcbf1d013d0e2e6f0c1f1/psutil-5.7.2-cp37-cp37m-linux_x86_64.whl Processing /home/lv-beast/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl Collecting requests-file==1.5.1 Using cached requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB) Collecting tqdm==4.50.2 Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB) Collecting vtk==8.1.2 Using cached vtk-8.1.2-cp37-cp37m-manylinux1_x86_64.whl (48.9 MB) Collecting icc_rt~=2019.0.0 Downloading icc_rt-2019.0-py2.py3-none-manylinux1_x86_64.whl (11.2 MB)

2020-10-22T04:39:48.540Z lv-beast:gpu0,1 DEBUG ERROR: Could not find a version that satisfies the requirement vs2015_runtime~=14.16.27012 (from -r /tmp/cached-reqskjekczdo.txt (line 20)) (from versions: none) ERROR: No matching distribution found for vs2015_runtime~=14.16.27012 (from -r /tmp/cached-reqskjekczdo.txt (line 20)) Command 'source /home/lv-beast/miniconda3/etc/profile.d/conda.sh && conda activate /home/lv-beast/.trains/venvs-builds.1/3.7 && pip install -r /tmp/cached-reqskjekczdo.txt' returned non-zero exit status 1.

trains_agent: ERROR: Could not install task requirements! Command 'source /home/lv-beast/miniconda3/etc/profile.d/conda.sh && conda activate /home/lv-beast/.trains/venvs-builds.1/3.7 && pip install -r /tmp/cached-reqskjekczdo.txt' returned non-zero exit status 1.

bmartinn commented 4 years ago

so first of all it worked! (when I launched from linux computer)

Yey!!!

when I try to launch from my dev (windows) it has conflicts (because of some windows lib that arent necessary for linux at all (like visualstudio things, that have another replacement for linux)

Yes, this is exactly why pip/conda freeze are ill-advised :)

The automatic conda package detection was also improved, hopefully it will just work out of the box (remember that we never actually tested it)

  1. Install the latest trains RC (just released with the conda improvements) on the windows development machine 
    pip install trains==0.16.3rc0
  2. Remove the sdk.development.detect_with_conda_freeze: true from the trains.conf file on the windows development machine
  3. Re-run the code
  4. On the trains-agent machine, make sure you have the latest trains-agent RC 
    pip install trains-agent==0.16.2rc0
  5. Clone and try to run on the trains-agent machine

:crossed_fingers:

tomeramit commented 4 years ago

I did all the steps,

trains==0.16.3rc0 on dev machine trains-agent==0.16.2rc0 on remote machine

unfortunately it didn't work, (somehow it did not recognize the astra as installed package)

this is the information from conda list of the astra package (if it's needed) astra-toolbox 1.9.0.dev11 np116py37_0 astra-toolbox/label/dev

2020-10-25T04:22:45.775Z lv-beast:gpu0,1 INFO task 5d29048bca244071a10d036c0c7b06f6 pulled from ca9acd7a32df49e5b4ea41760e252be7 by worker lv-beast:gpu0,1

2020-10-25T04:22:50.824Z lv-beast:gpu0,1 DEBUG Current configuration (trains_agent v0.16.2rc0, location: /tmp/.trainsagent.uih7n7c.cfg):

sdk.storage.cache.default_base_dir = ~/.trains/cache sdk.storage.cache.size.min_free_bytes = 10GB sdk.storage.direct_access.0.url = file://* sdk.metrics.file_history_size = 100 sdk.metrics.matplotlib_untitled_history_size = 100 sdk.metrics.images.format = JPEG sdk.metrics.images.quality = 87 sdk.metrics.images.subsampling = 0 sdk.metrics.tensorboard_single_series_per_graph = false sdk.network.metrics.file_upload_threads = 4 sdk.network.metrics.file_upload_starvation_warning_sec = 120 sdk.network.iteration.max_retries_on_server_error = 5 sdk.network.iteration.retry_backoff_factor_sec = 10 sdk.aws.s3.key = sdk.aws.s3.region = sdk.aws.boto3.pool_connections = 512 sdk.aws.boto3.max_multipart_concurrency = 16 sdk.log.null_log_propagate = false sdk.log.task_log_buffer_capacity = 66 sdk.log.disable_urllib3_info = true sdk.development.task_reuse_time_window_in_hours = 72.0 sdk.development.vcs_repo_detect_async = true sdk.development.store_uncommitted_code_diff = true sdk.development.support_stopping = true sdk.development.default_output_uri = sdk.development.force_analyze_entire_repo = false sdk.development.suppress_update_message = false sdk.development.detect_with_pip_freeze = false sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true sdk.development.worker.report_global_mem_used = false sdk.development.detect_with_conda_freeze = false agent.worker_id = lv-beast:gpu0,1 agent.worker_name = lv-beast agent.force_git_ssh_protocol = false agent.python_binary = agent.package_manager.type = conda agent.package_manager.pip_version = <20.2 agent.package_manager.system_site_packages = true agent.package_manager.force_upgrade = false agent.package_manager.conda_channels.0 = simpleitk agent.package_manager.conda_channels.1 = pytorch agent.package_manager.conda_channels.2 = astra-toolbox/label/dev agent.package_manager.conda_channels.3 = conda-forge agent.package_manager.conda_channels.4 = defaults agent.package_manager.conda_channels.5 = intel agent.package_manager.torch_nightly = false agent.venvs_dir = /home/lv-beast/.trains/venvs-builds.1 agent.vcs_cache.enabled = true agent.vcs_cache.path = /home/lv-beast/.trains/vcs-cache.1 agent.venv_update.enabled = false agent.pip_download_cache.enabled = true agent.pip_download_cache.path = /home/lv-beast/.trains/pip-download-cache agent.translate_ssh = true agent.reload_config = false agent.docker_pip_cache = /home/lv-beast/.trains/pip-cache agent.docker_apt_cache = /home/lv-beast/.trains/apt-cache.1 agent.docker_force_pull = false agent.default_docker.image = nvidia/cuda:10.1-runtime-ubuntu18.04 agent.git_user = tomer.amit agent.cuda_version = 101 agent.cudnn_version = 70 agent.default_python = 3.7 api.version = 1.5 api.verify_certificate = true api.default_version = 1.5 api.http.max_req_size = 15728640 api.http.retries.total = 240 api.http.retries.connect = 240 api.http.retries.read = 240 api.http.retries.redirect = 240 api.http.retries.status = 240 api.http.retries.backoff_factor = 1.0 api.http.retries.backoff_max = 120.0 api.http.wait_on_maintenance_forever = true api.http.pool_maxsize = 512 api.http.pool_connections = 512 api.api_server = http://192.168.2.94:8008 api.web_server = http://192.168.2.94:8080 api.files_server = http://192.168.2.94:8081 api.credentials.access_key = IQ4BUOO3Q27ZPICO096M api.host = http://192.168.2.94:8008

Executing task id [5d29048bca244071a10d036c0c7b06f6]: repository = https://bodyvisionmedical.visualstudio.com/LungVision/_git/MachineLearning branch = feature/add_amp version_num = 16b77c9b445dcc35b785f731df9d677f6663513c tag = docker_cmd = entry_point = training_script.py working_dir = LvObjects/Lv3D/TomoGan/scripts

Executing Conda: /home/lv-beast/miniconda3/condabin/conda env remove -p /home/lv-beast/.trains/venvs-builds.1/3.7 --quiet --json

2020-10-25T04:23:20.887Z lv-beast:gpu0,1 DEBUG Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults -c intel cudatoolkit=10.1 --quiet --json

2020-10-25T04:23:45.955Z lv-beast:gpu0,1 DEBUG Pass

Using cached repository in "/home/lv-beast/.trains/vcs-cache.1/MachineLearning.2f27b16d384a01d04c7eedd7180ff087/MachineLearning"

2020-10-25T04:23:50.993Z lv-beast:gpu0,1 DEBUG From https://bodyvisionmedical.visualstudio.com/LungVision/_git/MachineLearning

2020-10-25T04:23:56.027Z lv-beast:gpu0,1 DEBUG Note: switching to '16b77c9b445dcc35b785f731df9d677f6663513c'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example:

git switch -c

Or undo this operation with:

git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 16b77c9 add envs

2020-10-25T04:24:11.084Z lv-beast:gpu0,1 DEBUG type: git url: https://bodyvisionmedical.visualstudio.com/LungVision/_git/MachineLearning branch: HEAD commit: 16b77c9b445dcc35b785f731df9d677f6663513c root: /home/lv-beast/.trains/venvs-builds.1/3.7/task_repository/MachineLearning

Executing Conda: /home/lv-beast/miniconda3/condabin/conda install -p /home/lv-beast/.trains/venvs-builds.1/3.7 -c simpleitk -c pytorch -c astra-toolbox/label/dev -c conda-forge -c defaults -c intel 'pip<20.2' --quiet --json

2020-10-25T04:24:41.175Z lv-beast:gpu0,1 DEBUG Pass Conda: Trying to install requirements: ['certifi~=2020.6.20', 'chardet~=3.0.4', 'cryptography~=3.1.1', 'h5py~=2.10.0', 'imgaug~=0.4.0', 'joblib~=0.17.0', 'numba~=0.51.2', 'numpy~=1.16.5', 'pandas~=1.1.3', 'Pillow~=7.2.0', 'plotly~=4.11.0', 'protobuf~=3.13.0', 'pyclipper~=1.1.0', 'pytorch~=1.6.0', 'requests~=2.24.0', 'scikit_image~=0.17.2', 'scikit_learn~=0.23.2', 'scipy~=1.5.2', 'SimpleITK~=2.0.0', 'tensorboard~=2.3.0', 'torchvision~=0.7.0', 'urllib3~=1.25.10', 'graphviz', 'python-graphviz', 'kiwisolver'] Executing Conda: /home/lv-beast/miniconda3/condabin/conda env update -p /home/lv-beast/.trains/venvs-builds.1/3.7 --file /tmp/conda_envrxzirrbo.yml --quiet --json

2020-10-25T04:25:21.263Z lv-beast:gpu0,1 DEBUG Conda error: ResolvePackageNotFound:

2020-10-25T04:26:26.407Z lv-beast:gpu0,1 DEBUG Pass Conda: Installing requirements: step 2 - using pip: ['colorama==0.4.3', 'colorlog==4.4.0', 'dask==2.30.0', 'jsonpickle==1.4.1', 'log_utils==0.3.4', 'matplotlib==3.3.2', 'psutil==5.7.2', 'git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization', 'tqdm==4.50.2', 'trains==0.16.3rc0', 'vtk==8.1.2', 'scikit_learn~=0.23.2', 'scikit_image~=0.17.2'] bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash)

2020-10-25T04:26:31.445Z lv-beast:gpu0,1 DEBUG Collecting colorama==0.4.3 Using cached colorama-0.4.3-py2.py3-none-any.whl (15 kB) Collecting colorlog==4.4.0 Using cached colorlog-4.4.0-py2.py3-none-any.whl (14 kB) Requirement already satisfied: dask==2.30.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqspfiyp8gq.txt (line 3)) (2.30.0) Collecting jsonpickle==1.4.1 Using cached jsonpickle-1.4.1-py2.py3-none-any.whl (36 kB) Collecting log_utils==0.3.4 Using cached log_utils-0.3.4-py3-none-any.whl (11 kB) Requirement already satisfied: matplotlib==3.3.2 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqspfiyp8gq.txt (line 6)) (3.3.2) Processing /home/lv-beast/.cache/pip/wheels/2d/43/97/00701864a7bee6d9e1a52dd682537dcbf1d013d0e2e6f0c1f1/psutil-5.7.2-cp37-cp37m-linux_x86_64.whl Collecting tqdm==4.50.2 Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB) Collecting trains==0.16.3rc0 Downloading trains-0.16.3rc0-py2.py3-none-any.whl (795 kB) Collecting vtk==8.1.2 Using cached vtk-8.1.2-cp37-cp37m-manylinux1_x86_64.whl (48.9 MB) Collecting scikit_learn~=0.23.2 Using cached scikit_learn-0.23.2-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB) Requirement already satisfied: scikit_image~=0.17.2 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from -r /tmp/cached-reqspfiyp8gq.txt (line 13)) (0.17.2) Requirement already satisfied: pyyaml in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from dask==2.30.0->-r /tmp/cached-reqspfiyp8gq.txt (line 3)) (5.3.1) Requirement already satisfied: importlib-metadata in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from jsonpickle==1.4.1->-r /tmp/cached-reqspfiyp8gq.txt (line 4)) (2.0.0) Requirement already satisfied: cycler>=0.10 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqspfiyp8gq.txt (line 6)) (0.10.0) Requirement already satisfied: pillow>=6.2.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqspfiyp8gq.txt (line 6)) (7.2.0) Requirement already satisfied: certifi>=2020.06.20 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqspfiyp8gq.txt (line 6)) (2020.6.20) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqspfiyp8gq.txt (line 6)) (2.4.7) Requirement already satisfied: numpy>=1.15 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqspfiyp8gq.txt (line 6)) (1.16.5) Requirement already satisfied: kiwisolver>=1.0.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqspfiyp8gq.txt (line 6)) (1.2.0) Requirement already satisfied: python-dateutil>=2.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from matplotlib==3.3.2->-r /tmp/cached-reqspfiyp8gq.txt (line 6)) (2.8.1) Requirement already satisfied: plotly>=3.9.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (4.11.0) Collecting requests-file>=1.4.2 Using cached requests_file-1.5.1-py2.py3-none-any.whl (3.7 kB) Requirement already satisfied: six>=1.11.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (1.15.0) Collecting funcsigs>=1.0 Using cached funcsigs-1.0.2-py2.py3-none-any.whl (17 kB) Requirement already satisfied: attrs>=18.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (20.2.0) Requirement already satisfied: urllib3>=1.21.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (1.25.11) Requirement already satisfied: requests>=2.20.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (2.24.0)

2020-10-25T04:26:36.485Z lv-beast:gpu0,1 DEBUG Collecting humanfriendly>=2.1 Using cached humanfriendly-8.2-py2.py3-none-any.whl (86 kB) Processing /home/lv-beast/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl Collecting pathlib2>=2.3.0 Using cached pathlib2-2.3.5-py2.py3-none-any.whl (18 kB) Requirement already satisfied: pyjwt>=1.6.4 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (1.7.1) Collecting jsonschema>=2.6.0 Using cached jsonschema-3.2.0-py2.py3-none-any.whl (56 kB) Collecting furl>=2.0.0 Using cached furl-2.1.0-py2.py3-none-any.whl (20 kB) Requirement already satisfied: joblib>=0.11 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_learn~=0.23.2->-r /tmp/cached-reqspfiyp8gq.txt (line 12)) (0.17.0) Requirement already satisfied: scipy>=0.19.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_learn~=0.23.2->-r /tmp/cached-reqspfiyp8gq.txt (line 12)) (1.5.2) Collecting threadpoolctl>=2.0.0 Using cached threadpoolctl-2.1.0-py3-none-any.whl (12 kB) Requirement already satisfied: networkx>=2.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqspfiyp8gq.txt (line 13)) (2.5) Requirement already satisfied: imageio>=2.3.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqspfiyp8gq.txt (line 13)) (2.9.0) Requirement already satisfied: tifffile>=2019.7.26 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqspfiyp8gq.txt (line 13)) (2020.10.1) Requirement already satisfied: PyWavelets>=1.1.1 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from scikit_image~=0.17.2->-r /tmp/cached-reqspfiyp8gq.txt (line 13)) (1.1.1) Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from importlib-metadata->jsonpickle==1.4.1->-r /tmp/cached-reqspfiyp8gq.txt (line 4)) (3.3.2) Requirement already satisfied: retrying>=1.3.3 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from plotly>=3.9.0->trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (1.3.3) Requirement already satisfied: chardet<4,>=3.0.2 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (3.0.4) Requirement already satisfied: idna<3,>=2.5 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from requests>=2.20.0->trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (2.10) Processing /home/lv-beast/.cache/pip/wheels/a5/52/bf/71258a1d7b3c8cbe1ee53f9314c6f65f20385481eaee573cc5/pyrsistent-0.17.3-cp37-cp37m-linux_x86_64.whl Requirement already satisfied: setuptools in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from jsonschema>=2.6.0->trains==0.16.3rc0->-r /tmp/cached-reqspfiyp8gq.txt (line 10)) (49.6.0.post20201009) Collecting orderedmultidict>=1.0.1 Using cached orderedmultidict-1.0.1-py2.py3-none-any.whl (11 kB) Requirement already satisfied: decorator>=4.3.0 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from networkx>=2.0->scikit_image~=0.17.2->-r /tmp/cached-reqspfiyp8gq.txt (line 13)) (4.4.2) Installing collected packages: colorama, colorlog, jsonpickle, log-utils, psutil, tqdm, requests-file, funcsigs, humanfriendly, future, pathlib2, pyrsistent, jsonschema, orderedmultidict, furl, trains, vtk, threadpoolctl, scikit-learn Successfully installed colorama-0.4.3 colorlog-4.4.0 funcsigs-1.0.2 furl-2.1.0 future-0.18.2 humanfriendly-8.2 jsonpickle-1.4.1 jsonschema-3.2.0 log-utils-0.3.4 orderedmultidict-1.0.1 pathlib2-2.3.5 psutil-5.7.2 pyrsistent-0.17.3 requests-file-1.5.1 scikit-learn-0.23.2 threadpoolctl-2.1.0 tqdm-4.50.2 trains-0.16.3rc0 vtk-8.1.2 bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Replacing original pip vcs 'git+https://github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization' with 'git+https://tomer.amit:xxxxxx@github.com/BodyVisionMedical/pyserialization.git@ef992361a3a257cbd077c0306e0ad51342cc56ad#egg=pyserialization' bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Processing /home/lv-beast/.cache/pip/wheels/2f/e6/e4/fe5ef74318123672cef559a4cb281c4af196839a116f89c7fd/pyserialization-0.1.1-py3-none-any.whl Requirement already satisfied: jsonpickle in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from pyserialization) (1.4.1) Requirement already satisfied: numpy in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from pyserialization) (1.16.5) Requirement already satisfied: importlib-metadata in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from jsonpickle->pyserialization) (2.0.0) Requirement already satisfied: zipp>=0.5 in /home/lv-beast/.trains/venvs-builds.1/3.7/lib/python3.7/site-packages (from importlib-metadata->jsonpickle->pyserialization) (3.3.2) Installing collected packages: pyserialization Successfully installed pyserialization-0.1.1 bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Executing Conda: /home/lv-beast/miniconda3/condabin/conda list --json -p /home/lv-beast/.trains/venvs-builds.1/3.7

2020-10-25T04:26:41.517Z lv-beast:gpu0,1 DEBUG Running task id [5d29048bca244071a10d036c0c7b06f6]: [LvObjects/Lv3D/TomoGan/scripts]$ source /home/lv-beast/miniconda3/etc/profile.d/conda.sh && conda activate /home/lv-beast/.trains/venvs-builds.1/3.7 && /home/lv-beast/.trains/venvs-builds.1/3.7/bin/python -u training_script.py Summary - installed python packages: conda:

Environment setup completed successfully

bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Starting Task Execution:

bash: /home/lv-beast/miniconda3/envs/keras-dl/lib/libtinfo.so.5: no version information available (required by bash) Traceback (most recent call last): File "training_script.py", line 23, in from tomo.projectors.parallel_projection import ParallelProjection File "/home/lv-beast/.trains/venvs-builds.1/3.7/task_repository/MachineLearning/LungVision/tomo/tomo/projectors/parallel_projection.py", line 1, in import astra ModuleNotFoundError: No module named 'astra'