Reftable import and merge FAILED with exit code 1

[tylerdevos]

Hello, I am working with a large SNP dataset in DIYABC v2.1.0, and have run into an issue importing the reftable.bin file back into the GUI after generating it on a remote cluster. When I try to open the reftable through the "import and merge reftable file" option, nothing happens in the GUI. When I open the logfile after attempting to import the reftable, an error message is present that reads "Reftable import and merge FAILED with exit code 1". I have been unable to find any information in the DIYABC manual or elsewhere online regarding this error message, and thus have no idea how to resolve the issue. Could you let me know what the possible causes and solutions for this error message may be? Thanks so much for any information you can provide! Best, Tyler

[gdurif]

Hi,

We are not maintaining DIYABC v2.1.0 anymore. It has been replaced with DIYABC-RF (https://diyabc.github.io/) and especially the new graphical interface DIYABC-RF GUI which is implemented in this repository as a Shiny app (inside an R package). The new GUI is still a work in progress and some issues are currently being solved.

The new GUI offer you the possibility to import an existing project with an existing reftable file (at least it should work, if not, do not hesitate to report it here).

Best,

[tylerdevos]

Hi Ghislain,

Thank you for your response! I now have the new GUI up and running, and was able to import the old dataset, model, and reftable. However, a statobs.txt file does not seem to have been generated on the cluster where the reftable was produced, and this is preventing me from generating and viewing parameter estimates. Does the new version of DIYABC offer any way to generate the necessary statobsRF.txt file from an existing reftableRF.txt file, or would I need to regenerate all 1000000 simulations in order to get a corresponding statobs file?

Many thanks, Tyler

[gdurif]

Hi,

There will be no need to regenerate your 1000000 simulations. The idea is to call the simulation engine and asking to generate 0 simulations, and it will regenerate the statobsRF.txt file.

Sadly, it is not possible in the GUI at the moment (we have to fix that), and I'll explain why. Here is how you would expect to do it (if it worked).

You can import your project files in the new GUI including your header and reftable files (by the way, you will have to rename your old reftable file as reftableRF.txt for the interface to accept it). And then, you can try to generate 0 simulations (or a number lower than your current number of simulations in your reftable).

But problem: at the moment, the GUI will not start the simulation engine if the requested number of simulations is not higher than the current size of the existing reftable file... (I will try to modify this ASAP).

Another solution would be to use the command line interface. Would that be an option for you ?

Every thing is described here : https://diyabc.github.io/cli/

For instance, on my Linux machine (allowing the program to use 8 cores), I would run the following command in the project directory:

diyabc-RF-linux-v1.1.27 -p ./ -n "t:8"
diyabc-RF-linux-v1.1.27 -p ./ -R "ALL" -m -t 8 -g 8 -r 0

Best,

[gdurif]

Hi again,

I jumped to conclusion too fast. Maybe the existing reftable file will not be compatible with the new statobsRF.txt file.

Your reftable file was generated with the old version of the diyabc (v2.1.0) software, wasn't it? Since the new version (so called diaybc-RF), by default all possible summary statistics are computed and stored in the reftable file and statobs file. As a consequence, if you did use a subset of the summary statistics to construct your reftable file with the old version of the software, the new statobs file will not be compatible with your reftable file.

If you are in this case, I will investigate how we can solve this.

Best,

[tylerdevos]

Hi,

Thank you for looking into this. Command line is an option for me if necessary, but unfortunately I was only using 14 of the available summary statistics when the reftable was generated in v2.1.0--so it seems there will be a compatibility issue. If there is a way to solve this, I would be very grateful!

Best, Tyler

[gdurif]

Hi, sorry for my late answer. Do you still have the original header.txt file that was used to generate the ref table ?

If so, at the end, you should have the list of summary statistics that were used. You could specify them to the diyabc command line program with the -R option.

@fradav could you confirm this ? (thanks)

[tylerdevos]

Yes I do have the header.txt file. I'm not seeing any documented examples where the -R flag is set to anything other than "ALL"; could you advise me regarding the format in which the summary statistics should be specified after the flag? If it makes a difference, the 14 statistics I have in the header are as follows:

HP0 1 2 3
HM1 1 2 3
FP0 1&2 1&3 2&3
FM1 1&2 1&3 2&3
AP0 2&1&3
AM1 2&1&3

[gdurif]

Could you copy the final line of your header file ?

[gdurif]

I am really sorry. After doing some digging, it appears that I was wrong. The summary statistics changed between the old diyabc and the new diyabc-rf so you will not be able to use your reftable generated with diyabc.

In addition, the number of summary statistics managed by diyabc was too small to be used in diyabc-rf where the random forest analysis is interested with a huge number of summary statistics.

I am again really sorry for the false hope.

[fradav]

The final line in the header file contains all summary stat, after the parameters.

The -R option could specify classes of summary stats you want to include, separated by comma, they are documented in diyabc-RF’s user manual, “Summary statistics as components of the feature vector”.

But I doubt this could be of any use to your actual problem. Indeed, you got a pre-diyabcRF generated reftable, and as summary stats for SNP were redesigned since then, there is no one-to-one conversion, so computing statobs for a mismatched reftable is pointless.

However, since your actual statobs.txt does match your reftable, just copy/rename it to statobsRF.txt and you should be able to run new RF methods of analysis on it. There’s one caveat you must be aware of: the old set of summary statistics may not be sufficient to run meaningful sound and reliable RF posterior analysis, due to their lack of “expressiveness”, lack that the new and improved one is precîsely designed to rectify in the SNP case with RF setting. Of course, if your own prior knowledge of the scenarii allows you to assert that the set of summary stats that you already computed is meaningful/expressive enough, then it shouln’t be a problem.

[tylerdevos]

Thank you both for looking into this. Given all that you just explained, I think it will at this point be well worth the extra time for me to start from scratch and generate a new reftable with diyabc-RF instead of forcing the old reftable into the new version of the program and losing functionality along the way. Hopefully it will be smooth sailing from here! Thanks again for your time and assistance, it has been much appreciated!

Best, Tyler

[gdurif]

You are welcome.

To use the new diyabc-rf, you can either use the command line interface with your original header file, or use the graphical interface either with your original header file or you can reconfigure one.

To use the new GUI, if you are on Windows, a new release will be available next week (the current one is kind of buggy). If you are on MacOS or Linux, you can install the R package from here, it should already work.

Best,

[gdurif]

I will close this issue. Do not hesitate to open another one if you encounter any problem with diyabc-rf.