ECOS problem reopening not due to low sample coverage but update

[CeliaRodrigues]

In the other similar issues, you keep telling Freyja users that they get an ECOS error message because they have samples with low coverage. That is not the case: I will show you here the graphs I got before and after updating Freyja maybe in September on the same data. Before updating (top), I got nice graphs and lineage separation. After updating (bottom), I got graphs that are not informative and the error message:

/home/miniconda3/envs/freyja-env/lib/python3.11/site-packages/cvxpy/reductions/solvers/solving_chain.py:336: FutureWarning: Your problem is being solved with the ECOS solver by default. Starting in CVXPY 1.5.0, Clarabel will be used as the default solver instead. To continue using ECOS, specify the ECOS solver explicitly using the solver=cp.ECOS argument to the problem.solve method.

warnings.warn(ECOS_DEPRECATION_MSG, FutureWarning) /home/miniconda3/envs/freyja-env/lib/python3.11/site-packages/cvxpy/problems/problem.py:1403: UserWarning: Solution may be inaccurate. Try another solver, adjusting the solver settings, or solve with verbose=True for more information. warnings.warn( building mix/depth matrices

[joshuailevy]

Hi @CeliaRodrigues,

Thanks for your message. There's a handful of reasons that ECOS warnings and errors can be triggered. The most common warnings are the ones you've mentioned, the first of which (the FutureWarning) has been patched and will be part of subsequent releases (see: https://github.com/andersen-lab/Freyja/blob/1fa14df1ad2512cb50620cff4296d6df4107b5e7/freyja/sample_deconv.py#L166). The second is a solver convergence warning, indicating that cvxpy, the solver that freyja uses, has failed to converge nicely to a solution. Often times this occurs as a result of insufficient information in the raw data (e.g., because of low coverage) to differentiate between different lineages. However, this is not the only way that this can happen- for example, you can have perfect coverage, but if your sample mixture differs wildly from one comprised of known lineages, this can also lead to a solver warning, or even an actual error that leads to termination of the command.

If you'd be interested, we'd be happy to take a look at some of your data to see what's going on. Feel free to share it here or with me at jolevy@scripps.edu.

Josh

[CeliaRodrigues]

I am sending you a bam file by email.

[CeliaRodrigues]

Are there some steps I should take to install freyja like you have it installed because you got good data out of my bam file and I still get the ECOS error and different data from yours?

Your lineages in the summary column: EG.5: 0.44, Other: 0.195, XBB.1.5: 0.17,... My lineages in the summary column: Omicron: 0.755, Other: 0.195

[joshuailevy]

This looks like it will be fixed by a quick freyja update command. The summarized lineages depend on the state of the curated_lineages.json metadata file, which will be downloaded automatically (along with the lineage barcodes) when you run the update command.

[CeliaRodrigues]

I run freyja update every time before running the pipeline, and it does not fix it. I just did it right now, still get the ECOS error, and the conda install shows me the version 1.3.12 instead of the most recent one 1.4.8.

[joshuailevy]

Hm... freyja update will not update the freyja package version, it'll only update the barcode/metadata files. Since you've installed freyja with conda, you'll have to update the package with conda as well. I don't think this should effect your particular problem though- that part of the codebase is pretty much unchanged since 1.3.12

[CeliaRodrigues]

I updated conda and re-installed freyja and I still get the same errors and different results from you. So could you tell me how to make freyja through github please? I manage to git clone, but makefile or other attempts are not letting me run the github freyja version.

[joshuailevy]

Before we go to a local build/install, can you send the metadata file (curated_lineages.json) that you're using? It seems to be out of date. This is used to build the summarized section of the output. The lineages and corresponding frequencies you get in your example output look fine, it seems to just be a summarization issue. You may also just want to try using the current one: https://github.com/andersen-lab/Freyja/blob/main/freyja/data/curated_lineages.json

[CeliaRodrigues]

curated_lineages.json This is the one I have. Now that I found where it is stored, I could just replace it with the one from github?

[CeliaRodrigues]

I replaced the lineage file into my conda folder but it did not change my output. Either this is not the problem or I did not copy it to the right location.

[joshuailevy]

You should be able to supply it directly to demix using the --meta option to make sure. Doesn't need to be placed into the freyja/data directory.

[CeliaRodrigues]

For some reason that did not work either. I ended up cloning the main repository into the environment and installing it over the conda module. That gave me the results I was looking for with the updated lineages. You can close this issue.