search

visual-layer / fastdup

fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and video datasets. It helps enhance the quality of both images and labels, while significantly reducing data operation costs, all with unmatched scalability.
Other
1.59k stars 77 forks source link

[Bug]: Failed when used docker cpuset #300

Closed wangxiong172086864 closed 8 months ago

wangxiong172086864 commented 10 months ago

What happened?

When docker using cpuset, onnxruntime can not set the session option inter_op_num_threads properly, then the fastdup run can't not get the results.

What did you expect to see?

No response

What version of fastdup were you runnning on?

1.44

What version of Python were you running on?

Python 3.7

Operating System

CentOS

Reproduction steps

No response

Relevant log output

No response

Attach a screenshot [Optional]

No response

Contact Details [Optional]

No response

dbickson commented 10 months ago

Hello @wangxiong172086864 we will be happy to help but we need some more detailed information. You are running on centos, but what docker OS are you using? We do have native Centos 7 build for fastup you can install directly from our release page. In addition send us the full error message you see. How are you using cpuset? If you are running without it does fastdup run? Thanks

wangxiong172086864 commented 10 months ago

Hello @wangxiong172086864 we will be happy to help but we need some more detailed information. You are running on centos, but what docker OS are you using? We do have native Centos 7 build for fastup you can install directly from our release page. In addition send us the full error message you see. How are you using cpuset? If you are running without it does fastdup run? Thanks

Hi, docker os is ubuntu 18.04.1 LTS. We using cpuset like this --cpuset-cpus="0-1,10-13,18-29,46-83", we can't catch the error message, the process stop at fd.run() method. I have tried to find the reason for this, I found someone said that onnxruntime may set sess_options.intra_op_num_threads an improper value when cpuset is used., but I can't figure it out since I can't change the intra_op_num_threads value. running the code without cpuset or just set cpuset in a continuous interval will be ok.

dbickson commented 10 months ago

Under the hood it is possible to change the intra_op_num_threads value using 0.2 API: fastdup.run(intput_dir, work_dir, turi_param='num_onnx_intra_threads=1') however the default is 1 anyway and I don't think it will help.. Maybe try to give the cpu opst a continuous range?

wangxiong172086864 commented 10 months ago

Under the hood it is possible to change the intra_op_num_threads value using 0.2 API: fastdup.run(intput_dir, work_dir, turi_param='num_onnx_intra_threads=1') however the default is 1 anyway and I don't think it will help.. Maybe try to give the cpu opst a continuous range?

Just tried fastdup.run(intput_dir, work_dir, turi_param='num_onnx_intra_threads=1') and still get stuck, we run the code in the environment where cpu opst can't be decided by users.

dbickson commented 8 months ago

Closing