search

discoproject / disco

a Map/Reduce framework for distributed computing
http://discoproject.org
BSD 3-Clause "New" or "Revised" License
1.63k stars 241 forks source link

ddfs gc may fail if we are in the middle of creating a tag #533

Open pooya opened 10 years ago

pooya commented 10 years ago

In order to reproduce this issue we can use the following for loop in shell: for i in $(seq 1000); do ddfs chunk chek$i ./xa? ; done

On a single node cluster.

If we start the gc from the UI now while this for loop is executing, the gc sometimes fails with something like the following error message: [error] GC: stopping, unable to get tag <<"chek617">>: {{missing,notfound},false}

This seems to be a race condition between creating the tag and building the map for ddfs gc. This is not a critical issue because the gc succeeds in the future invocations.

joowade commented 10 years ago

Hi,Im a new comer for DISCO.I want to get some help from you.I have pushed the data to a tag(ddfs chunk data:\ ./),even I can get the information about where they are located(ddfs blobs data:).But when I run the job , the stages including map ,map_shuffle,reduce, are 0 pending,0 waiting,0 running,0 done,0 failed.I think maybe the ddfs can't discriminate the tag?What's the caution?