search

seung-lab / igneous

Scalable Neuroglancer compatible Downsampling, Meshing, Skeletonizing, Contrast Normalization, Transfers and more.
GNU General Public License v3.0
43 stars 17 forks source link

Do I need to manually clear the folder where QUEUE is stored? #163

Closed liuyx599 closed 9 months ago

liuyx599 commented 9 months ago

I tried running the following command, I've noticed that this QUEUE folder is taking up a lot of space on hard drive, should I use the rm -rf QUEUE/* command to clean it up before each script use?

On the one hand I'm worried about the space it takes up as I use the script more and more, and on the other hand I'm worried about all the commands (e.g. generating mesh and skeleton all in the same QUEUE directory, and whether there will be any conflicts), but of course if igneous could handle such doubts automatically I might not have to worry so much.

As we all know, rm -rf is a very dangerous command that I wouldn't dare to use on a server cluster

rm -rf $QUEUE/*   # if needed?
igneous mesh forge $SEGPATH --dir mesh  --mip 2 --queue $QUEUE --sharded
igneous --parallel 8 execute $QUEUE
igneous mesh merge-sharded $SEGPATH --queue $QUEUE
igneous execute $QUEUE
william-silversmith commented 9 months ago

Hi!

Yes, it is advisable to clear the queue when it is completed. Instead of using rm -rf you can use the safer ptq purge DIRECTORY command that comes installed with igneous. ptq stands for "python task queue".

https://github.com/seung-lab/python-task-queue/

liuyx599 commented 9 months ago

i got it, thank you very much!