search

mar-file-system / marfs

MarFS provides a scalable near-POSIX file system by using one or more POSIX file systems as a scalable metadata component and one or more data stores (object, file, etc) as a scalable data component.
Other
96 stars 27 forks source link

Garbage Collection Packed File Error #170

Closed atorrez closed 7 years ago

atorrez commented 7 years ago

When marfs_gc enters the section of code that evaluates packed objects, I am seeing a grep write error during a pipes popen call with a (grep | wc -l) command.

One item to note is that the tmp_packed_log on cc-fta01 is substantially larger now versus previous runs of garbage collection. The initial thought is too many file handles are open at once.

atorrez commented 7 years ago

I have fixed the issue. It was related to number of file handles as Will suggested. It turns out I had a code path that was not closing pipes and with a long list of packed files it was running out of handles.

I am having an issue pushing the code back into trunk at the moment. I will try again later - I am off today and have an appt in S.F.

atorrez commented 7 years ago

Code pushed to master.