search

brdav / atrc

Exploring Relational Context for Multi-Task Dense Prediction [ICCV 2021]
MIT License
46 stars 3 forks source link

multitasking performance metrics #5

Closed 187185537 closed 1 year ago

187185537 commented 1 year ago

Hello, is there no multitasking performance metrics ∆m in this code?I can't find it.

brdav commented 1 year ago

Hi, no I didn't use code for that, I separately calculated the multitasking performance in a Google sheet.

187185537 commented 1 year ago

Did you use the two-step test results shown below to calculate your multitasking performance？ 1、python ./src/main.py --cfg ./config/nyud/hrnet18/atrc.yaml --model.atrc_genotype_path $GENOTYPE_DIR/atrc_genotype.json --datamodule.data_dir $DATA_DIR --trainer.gpus 1 2、python ./src/main.py --cfg ./config/nyud/hrnet18/baselinemt.yaml --datamodule.data_dir $DATA_DIR --trainer.gpus 1

brdav commented 1 year ago

Almost, the multi-task performance is calculated w.r.t. the single-task performance, so 2. should be: python ./src/main.py --cfg ./config/nyud/hrnet18/baselinest.yaml --datamodule.data_dir $DATA_DIR --trainer.gpus 1 for each task separately.

187185537 commented 1 year ago

sorry,My understanding is that
the results of single task from 1、python ./src/main.py --cfg ./config/nyud/hrnet18/atrc.yaml --model.atrc_genotype_path $GENOTYPE_DIR/atrc_genotype.json --datamodule.data_dir $DATA_DIR --trainer.gpus 1 the result of multi-task from 2、python ./src/main.py --cfg ./config/nyud/hrnet18/baselinemt.yaml --datamodule.data_dir $DATA_DIR --trainer.gpus 1 Am I getting that right？?

187185537 commented 1 year ago

Hello, can you tell me what the output results of each of these three steps mean? 1、python ./src/main_search.py --cfg ./config/nyud/hrnet18/atrc_search.yaml --datamodule.data_dir $DATA_DIR --trainer.gpus 2 --trainer.accelerator ddp 2、python ./src/main.py --cfg ./config/nyud/hrnet18/atrc.yaml --model.atrc_genotype_path $GENOTYPE_DIR/atrc_genotype.json --datamodule.data_dir $DATA_DIR --trainer.gpus 1 3、python ./src/main.py --cfg ./config/nyud/hrnet18/baselinemt.yaml --datamodule.data_dir $DATA_DIR --trainer.gpus 1 thanks

brdav commented 1 year ago
  1. This is doing the architecture search, so the output will be the network topology / genotype.
  2. This is for training a previously found genotype (with step 1.), so the output is the trained ATRC network.
  3. This is a simple multi-task baseline network, which uses a shared encoder and separate decoders for each task.