`auspice view` not showing radial or rootless tree

aofarrel commented 4 months ago

Current Behavior

While attempting to view a small tree in radial view, the radial tree never loads, even though the tree is small and it works fine in rectangular view. The developer console is full of hundreds of errors. A similar thing seems to happen in unrooted trees.

A possibly separate issue is preventing me from loading the same tree in nwk format into auspice, even though auspice.us supports nwk format.

Tree in json format --> ash_10samps.json Tree in nwk format (zipped due to GitHub restrictions) --> 10samps.nwk.zip

local auspice

The json version of the tree can be loaded into the viewer, but radial view does not work
The nwk version of the tree cannot be loaded into the viewer (doesn't even show up as a dataset)

auspice.us

The json version of the tree can be loaded into the viewer, but radial view does not work
The nwk version of the tree can be loaded into the viewer and radial view does work (this is expected behavior)

Expected behavior

If the json is valid, radial view should work
If the json is invalid or a reference sequence is needed somewhere, auspice should at some point stop throwing errors that only show up in the js console, and instead show some kind of warning to the user either on the terminal or in the browser window
Local!auspice should be able to see a valid nwk file in the working directory and use it as a dataset

How to reproduce locally

Steps to reproduce the current behavior:

Start with this tree: ash_10samps.json
auspice view
Browse to localhost:4000 and load the .json tree as normal

Note that [warning] Failed to read /Users/aofarrel/github/tree_nine/ash_10samps_root-sequence.json appears in the terminal when loading the tree, but from what I've seen online this shouldn't be an issue.

Confirm that the rectangular view works fine
Click "radial"
The tree-containing part of the screen goes blank and the js console files with errors
Confirm rootless tree also does not work, but the scatter plot does

How to reproduce on auspice.us

Drag and drop one of the trees into https://auspice.us/ as normal and click the radial view button.

Possible solution

The tree in question was generated from a .pb tree using matUtils, and I'm not seeing other people with similar issues, so this could well be an issue on matUtils' end. However, I'm not sure of a scenario where the json would be corrupted such that radial and unrooted don't work, but rectangular and scatter are fine. It also doesn't explain why local!auspice doesn't seem to see nwk files in the workdir.

Your environment: if running Nextstrain locally

Operating system: Mac OS
Browser: Safari and Firefox (tried both)
Version (e.g. auspice 2.7.0): 2.51.0

Additional context

The attached tree is tuberculosis data from SRA. Reference genome is H37Rv.
This was tested on several TB trees of different sizes and behavior was consistent across all of them.
https://github.com/yatisht/usher/issues/303 shouldn't be the issue here as this is a 10 sample tree and no metadata was loaded alongside it, and the issue persists whether or not metadata is attached.

jameshadfield commented 4 months ago

Thanks for the detailed bug report! The live site is working fine (e.g. https://nextstrain.org/zika?l=radial, Firefox) so I suspect it's something related to this specific dataset. I'll take a look next week if no-one's gotten there first.

aofarrel commented 4 months ago

It turns out this issue also affects auspice.us, albeit in a slightly different way. I've updated the first post with new information.

aofarrel commented 4 months ago

Having spoken with some others at UCSC, we're thinking the issue is sort of bidirectional. Auspice appears to expect certain extra data in a json that it does not expect in a newick file, at least when it comes to radial view. matUtils is probably not including that extra data when converting to json, but since auspice.us's radial view works on the much simpler nwk format, that data shouldn't be strictly necessary for radial view.

jameshadfield commented 4 months ago

The "wrapper" node in ash_10samps.json is missing a divergence value which is what's causing this bug. The following diff fixes the dataset, although you may want to use a different divergence value:

-     "name": "wrapper"
+     "name": "wrapper",
+     "node_attrs": {
+       "div": 0
+     }

We do have dataset validation tools in augur which let you know this dataset doesn't conform to the schema, however if often requires a little more digging to work out what's actually wrong (and there are some false-positives - for instance the validation also fails because of a missing meta.updated field, but Auspice works ok without this).

$ augur validate export-v2 ash_10samps.json
Validating schema of 'ash_10samps.json'...
  .tree {"children": [{"branch_attrs": {"labels": {"id":…} failed oneOf validation for [{"$ref": "#/$defs/tree"}, {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/tree"}}]
    validation for arm 0: {"$ref": "#/$defs/tree"}
      .tree {"children": [{"branch_attrs": {"labels": {"id":…} failed required validation for ["name", "node_attrs"]
    validation for arm 1: {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/tree"}}
      .tree {"children": [{"branch_attrs": {"labels": {"id":…} failed type validation for "array"
  .meta {"colorings": [{"key": "country", "title": "Coun…} failed required validation for ["updated", "panels"]
FATAL ERROR: Validation of 'ash_10samps.json' failed.

I've made this Augur issue to hopefully improve this output to be more useful.

Local!auspice should be able to see a valid nwk file in the working directory and use it as a dataset

Auspice itself can only use phylogenies in the Auspice JSON format. Auspice.us is a customised version of Auspice which adds the ability to read newick trees (as well as adding a drag-and-drop interface etc).

I'm not sure of a scenario where the json would be corrupted such that radial and unrooted don't work, but rectangular and scatter are fine

Ideally Auspice would raise a more sensible error (there's nothing in the browser console either). I'll keep this issue open as a reminder.

aofarrel commented 4 months ago

The "wrapper" node in ash_10samps.json is missing a divergence value which is what's causing this bug.

Correct me if I'm wrong, but doesn't the nwk formatted file lack that value too? What's throwing me off is that it seems like it's okay for a radial tree to be missing some data if it's in nwk format, but that same data being missing in json format is a problem. (I'm not super knowledgeable on the precise details of tree formats so please let me know if I'm way off base here.)

Local!auspice should be able to see a valid nwk file in the working directory and use it as a dataset

Auspice itself can only use phylogenies in the Auspice JSON format. Auspice.us is a customised version of Auspice which adds the ability to read newick trees (as well as adding a drag-and-drop interface etc).

Thank you for clarifying this, I wasn't aware of the distinction.

Also, I want to note that it turns out nwk files sometimes break in radial view on auspice.us too, although I've only seen this on very small nwk files. I can't share the nwk as it's protected data but will continue to poke around at it -- if I can something I can replicate with open data, I'll make a new ticket specific to the nwks not working in radial view on auspice.us.

jameshadfield commented 4 months ago

Correct me if I'm wrong, but doesn't the nwk formatted file lack that value too? What's throwing me off is that it seems like it's okay for a radial tree to be missing some data if it's in nwk format, but that same data being missing in json format is a problem.

For Auspice the JSON should adhere to the Auspice dataset schema, which requires every node to have a node_attrs object and them to have div or num_date properties set there (or both!). For auspice.us we add an extra layer of functionality onto Auspice by adding a newick to Auspice JSON dataset parser and this parser initializes nodes with node_attrs: {div: 0}, which probably explains the behaviour you're seeing.

Why it breaks in radial views and not rectangular views is somewhat unexpected, but ultimately it's because the code is expecting data in a format that it's not in. We should improve auspice here to detect the dataset's not correct and raise an error/warning.

nextstrain / auspice