fastdup is a powerful free tool designed to rapidly extract valuable insights from your image & video datasets. Assisting you to increase your dataset images & labels quality and reduce your data operations costs at an unparalleled scale.
Other
1.51k
stars
74
forks
source link
[Bug]: fastdup fails to create above 10M object crops on ubuntu 20 (due to file system exhaustion) #307
Storing crop images and subfolder fails to do file system exhaustion on ubuntu 20
Jan 7 00:00:03 ip-172-31-32-254 systemd[1]: Finished Daily man-db regeneration.
Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946205] EXT4-fs warning: 110 callbacks suppressed
Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946211] EXT4-fs warning (device nvme0n1p1): ext4_dx_add_entry:2544: Directory (ino: 271020531) index full, reach max htree level :2
Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946217] EXT4-fs warning (device nvme0n1p1): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem
dbickson
commented
5 months ago
What happened?
Storing crop images and subfolder fails to do file system exhaustion on ubuntu 20 Jan 7 00:00:03 ip-172-31-32-254 systemd[1]: Finished Daily man-db regeneration. Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946205] EXT4-fs warning: 110 callbacks suppressed Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946211] EXT4-fs warning (device nvme0n1p1): ext4_dx_add_entry:2544: Directory (ino: 271020531) index full, reach max htree level :2 Jan 7 00:00:06 ip-172-31-32-254 kernel: [824165.946217] EXT4-fs warning (device nvme0n1p1): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem
What did you expect to see?
No response
What version of fastdup were you runnning on?
1.82
What version of Python were you running on?
Python 3.9
Operating System
ubuntu 20
Reproduction steps
No response
Relevant log output
No response
Attach a screenshot [Optional]
No response
Contact Details [Optional]
No response