Issue in rficlean parallelization?

[lanky441]

I have one query. While the rficlean is running with parallelization, the first 15 parts are of very small sizes (in MBs) and part16 is basically the size of whole raw data. And I think this is making rficlean run very slow.

[lanky441]

Here is an example. The first 15 rficlean.fil files are around 65 MBs and rfiClean_part16.fil is already 9.5 GB. And according to my experience, part16 will contain the rest of the data too.

[ymaan4]

First it needs to be resolved whether it is happening due to (a) some recent change in the pipeline, or (b) a problematic formation of the rficlean command in the pipeline, or (c) a bug in the parallelization script (on rfiClean side). (a) and (b) can be resolved by running the same data (with same input parameters) through the previous version of the pipeline and looking through the logs

Since this issue has been noticed only recently, and it never came up in the previous version of the pipeline (it should have also come up in the tests that were done for the paper, but didn't), I tend to think that the issue is on the pipeline's side. Once it is proved that the issue is not in the pipeline, I am happy to look into rfiClean to see what is happening (I would need all the necessary information to reproduce the issue, so please share that already).

On Thu, Jul 23, 2020 at 10:29 PM Lankeswar Dey notifications@github.com wrote:

[image: rficlean_fil] https://user-images.githubusercontent.com/38290475/88335393-df8f2700-cd50-11ea-91ee-c140ac698abc.png

Here is an example. The first 15 rficlean.fil files are around 65 MBs and rfiClean_part16.fil is already 9.5 GB. And according to my experience, part16 will contain the rest of the data too.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/abhisrkckl/pinta/issues/12#issuecomment-663216755, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXWLAFBE67LZ54R2RALOMLR5CMUNANCNFSM4PF7ISSA .

[abhisrkckl]

@lanky441 Can you point me to the dataset where you observed this issue?

[shantanu9847]

From anecdotal evidence, it is happening for all data. So you can pick any data you want.

[lanky441]

@abhisrkckl For example, you can take the data present in /Data/bcj/INPTA/38_014/01June2020_band3.

I have analyzed this data and pipeline.in & log files are present in /Data/lankeswar.dey/Analysis/38_014/01Jun/band3

[abhisrkckl]

I tested this with the older version of the pipeline and it seems to work fine. The issue is specific to the new version.

[abhisrkckl]

One major difference I can see between the two versions is that the new version accesses the file through a symlink whereas the older version accesses the file directly.

I tried running the rficlean command from the new version replacing the symlinks with the actual path and it works without problems. So this is likely related to the new version's use of symlinks.

[abhisrkckl]

Possible cause:

The crp_rficlean_gm.sh uses the "stat -c%s" command to find the size of the raw data file. But when the input is a symlink, this gives the wrong result. The -L option can dereference the symlink and get the real file size.

Eg: bash-4.2$ stat -c%s J1857+0943.25august2018.band5cdp.dat.dat 78 bash-4.2$ stat -Lc%s J1857+0943.25august2018.band5cdp.dat.dat 36003905536

But still doesn't explain how the 64MB size of the smaller files comes about.

[ymaan4]

64 MB is explainable. I introduced "-L", can you try now?

On Fri, Jul 24, 2020 at 1:01 PM Abhimanyu Susobhanan < notifications@github.com> wrote:

Possible cause:

The crp_rficlean_gm.sh uses the "stat -c%s" command to find the size of the raw data file. But when the input is a symlink, this gives the wrong result. The -L option can dereference the symlink and get the real file size.

Eg: bash-4.2$ stat -c%s J1857+0943.25august2018.band5cdp.dat.dat 78 bash-4.2$ stat -Lc%s J1857+0943.25august2018.band5cdp.dat.dat 36003905536

But still doesn't explain how the 64MB size of the smaller files comes about.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/abhisrkckl/pinta/issues/12#issuecomment-663485147, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXWLAHKJJFRPUYXVBZD7KLR5FS2NANCNFSM4PF7ISSA .

[ymaan4]

64 MB comes because, by design, at least 1 block is processed per file, and for the choice of parameters, it will exactly be 64 MB. BTW, quite an interesting issue and nice find, I think it should work now.

On Fri, Jul 24, 2020 at 1:04 PM Yogesh Maan ymaan4@gmail.com wrote:

64 MB is explainable. I introduced "-L", can you try now?

On Fri, Jul 24, 2020 at 1:01 PM Abhimanyu Susobhanan < notifications@github.com> wrote:

Possible cause:

The crp_rficlean_gm.sh uses the "stat -c%s" command to find the size of the raw data file. But when the input is a symlink, this gives the wrong result. The -L option can dereference the symlink and get the real file size.

Eg: bash-4.2$ stat -c%s J1857+0943.25august2018.band5cdp.dat.dat 78 bash-4.2$ stat -Lc%s J1857+0943.25august2018.band5cdp.dat.dat 36003905536

But still doesn't explain how the 64MB size of the smaller files comes about.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/abhisrkckl/pinta/issues/12#issuecomment-663485147, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXWLAHKJJFRPUYXVBZD7KLR5FS2NANCNFSM4PF7ISSA .

[abhisrkckl]

With this change, it seems to work fine. @shantanu9847 @lanky441 Please check.

[shantanu9847]

Is the latest version which fixes this updated on tapti at /misc/home/asusobhanan/Work/V5/pinta? (if not let me know where the pinta executable which fixes this is )

[abhisrkckl]

Yes

[ymaan4]

I think this issue was fully resolved, hence closing. If that was not the case, then please re-open.

[abhisrkckl]

Yes.. I forgot to close this issue earlier.