problems with CSV file merging

[bcdarwin]

Depending on combinations of pipeline (MBM vs two-level), --{no-,}run-maget setting, --files/--csv-file, etc., various crashes are possible (see also #382 for more issues). Overall the problem seems to be due to conditionally present columns in data frames. I propose the following fixes:

• get_imgs be generalized to handle both cases by returning additional CSV columns so that (1) group IDs are available to the two-level pipeline and other input metadata (including file, group) is available to be written out to 'analysis.csv'.
• get_imgs defaults to filenames relative to the CSV file but can optionally handle filenames relative to the pipeline (via a flag). Open to adopting the opposite convention.
• the additional _merge columns be removed from the code since they both cause breakage (as they are only conditionally present!) and potentially wrong (is the basename really always sufficient to identify a file???) and actual paths be used instead.

[bcdarwin]

So, get_imgs probably doesn't need to be changed. We can join its result with the full input CSV provided the file paths in both are correctly relativized (should make a procedure for this?). The default should be that the paths in a CSV file are relative to the CSV (since then only one CSV is needed per dataset rather than per pipeline), but a flag should be provided to recover the other behaviour of pipeline-relative paths since this might be useful and is what the two-level pipeline currently does.

[gdevenyi]

Just bitten by this for a --bootstrap with no maget or masking. Will test.

[gdevenyi]

develop seems to work, had to figure out how to install xfmavg_scipy.py :)

[bcdarwin]

I have a PR to fix the installation ....

---

From: Gabriel A. Devenyi notifications@github.com Sent: December 19, 2018 3:48:41 PM To: Mouse-Imaging-Centre/pydpiper Cc: Ben Darwin; Author Subject: Re: [Mouse-Imaging-Centre/pydpiper] problems with CSV file merging (#413)

develop seems to work, had to figure out how to install xfmavg_scipy.py :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Mouse-2DImaging-2DCentre_pydpiper_issues_413-23issuecomment-2D448738624&d=DwMCaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=LkJZp0VpGAxKM5H8kprmlug3pH1d4b6iqxJoh49Xl8k&s=ixhjp5mx9ttWmZsck3xVNr3QGDBXTOPJ6aD-DHAElHQ&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAKBNFQ4FGqBKCwYQBveQmQeV1pmssOJks5u6qYpgaJpZM4Yv8lN&d=DwMCaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=LkJZp0VpGAxKM5H8kprmlug3pH1d4b6iqxJoh49Xl8k&s=RFcpVl4TzjmskImYcYRfoNq8I9vgkDCK_5zysU0tJpM&e=.

---

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.

[gdevenyi]

ah, interesting, I was installing pyezminc rather than minc2-simple, both seem to have xfmavg_scipy.py