search

huawei-noah / vega

AutoML tools chain
http://www.noahlab.com.hk/opensource/vega/
Other
845 stars 176 forks source link

can user set worker_path in config? #88

Closed Lvhhhh closed 3 years ago

Lvhhhh commented 3 years ago

evey time you run pipline.py . the worker_path is random generated to save model and tensorboard log. such as 0126.091415.578 and 0126.090933.510 etc. it is very inconvenient. can you set the fixed worker_path in yml txt？ in addition, during one task - training , i found some parameter set in yml and some parameter set in config py. it is very informal. as a big company . i think you should Specificate code framework. tks

zhangjiajin commented 3 years ago

Hi @Lvhhhh , Can you give an example of which configurations are partially in the yml file and partially in the config.py file? When using Vega, you are advised to adjust the configuration in the yml file instead of the config.py file in the source code.

Lvhhhh commented 3 years ago

cyclesr.yml crop_size is from zeus/datasets/conf/div2k.py

Lvhhhh commented 3 years ago

and can we write the worker_path in yml file? every time i rename the random generated worker_path...

zhangjiajin commented 3 years ago

You only need to modify the YAML configuration file. For example:

    dataset:
        type: Div2kUnpair
        common:
            upscale: 4
        train:
            HR_dir: "/cache/datasets/DIV2K_unpair/DIV2K_HR_sub480"
            LR_dir: "/cache/datasets/DIV2K_unpair/DIV2K_unknown_sub120"
            batch_size: 64
            crop_size: 120
        test:
            HR_dir: "/cache/datasets/DIV2K_unpair/DIV2K_valid_HR"
            LR_dir: "/cache/datasets/DIV2K_unpair/DIV2K_valid_LR_unknown"
            batch_size: 1
zhangjiajin commented 3 years ago

When Vega starts a pipeline, it creates a directory containing time because some files are temporarily cached in the directory. Do not specify a directory to ensure the reliability of the training process. You mentioned the need to rename the directory. Could you describe your application scenario? After the training is complete, directly obtain the result from this directory. Why do you need to rename the directory?

Lvhhhh commented 3 years ago

When Vega starts a pipeline, it creates a directory containing time because some files are temporarily cached in the directory. Do not specify a directory to ensure the reliability of the training process. You mentioned the need to rename the directory. Could you describe your application scenario? After the training is complete, directly obtain the result from this directory. Why do you need to rename the directory?

thank you for your answer. i need to rename the directory because i do many experiments at the same time. how can i distinguish which directory correspondence to the experiments i wanted?

zhangjiajin commented 3 years ago

Please try the following configurations (cyclesr.yml):

general:
    task:
        local_base_path: "/base_dir"
        task_id: "task_id_dir"

pipeline: [fully_train]

The task directory will be fixed as "/base_dir/task_id_dir".

Lvhhhh commented 3 years ago

Please try the following configurations (cyclesr.yml):

general:
    task:
        local_base_path: "/base_dir"
        task_id: "task_id_dir"

pipeline: [fully_train]

The task directory will be fixed as "/base_dir/task_id_dir".

ok！thank you