itolexport.py unable to export large tree

[Mestizia]

Hi,

First of all, thanks for the work on Itol, it's a really nice software to visualize tree data and I like how easy it is to add annotation by just dropping text files on it, brilliant! I was trying to export a large [18k] annotated tree both through the website and the command line. However, the tree is missing part of the annotation [the bars are not present but it shows the barplot scale]. Everything works fine with a smaller tree with the same kind of plot. Is there a specific command I can use to force to export the complete tree? Also, you mention in README.rst that the flag to run the command in verbose is -r while it should be -v; which by the way does not seem to be working.

Many thanks,
V

[albertyw]

I updated the verbose flag in 70d708f568718beb96eedc483d2a15f4a59b61e5 but the missing annotations is probably from itol's server itself. I've had luck with using a different export format which may fix annotations.

[jeffreyjeromequinn]

Hi Albert,

Thanks for your work on this API! I'm having an issue that may be related to Mestizia's above regarding large trees.

I too am trying to use the ItolExport() function to export large trees (100s - 10,000s leaves) with a large number of visible datasets (~30 - 80 "color strips"). I am able to export some of the smaller trees without issue, but when I try with the larger ones I get an empty output image file. Notably, this issue is independent of image file format (e.g. SVG, PNG, EPS, etc.). Also, if I limit the number of visible datasets, it will successfully export the image file.

Do you know what could be causing this?

Cheers, Jeff

[albertyw]

Can you attach any sample data/configs?

[jeffreyjeromequinn]

Hi Albert, Attached is one (relatively small) example, including a tree file (*.tree.txt) and 34 color strip files. Let me know if you'd like any other information. Thanks for looking into this. Best, Jeff LG19_iTOL.zip

[albertyw]

I was able to reproduce this with

"""
Before running this script, run
wget https://github.com/albertyw/itolapi/files/3924504/LG19_iTOL.zip
unzip LG19_iTOL.zip
"""
from itolapi import Itol, ItolExport

tree_file = 'LG19_iTOL/LG19_tree.processed.tree.txt'
colors = ['LG19_iTOL/indelColors_%s.txt' % ('0'+str(i))[-2:] for i in range(33)]
colors.append('LG19_iTOL/cellSampleID.txt')
export_location = 'asdf.svg'
dataset_count = 33
# setting dataset_count to 1 will generate a valid image

"""
client = Itol()
client.add_file(tree_file)
client.params['treeName'] = 'Example'
for color in colors:
    client.add_file(color)
client.print_variables()
status = client.upload()
print('status: %s' % status)
print('id: %s' % client.comm.tree_id)
print('output: %s' % client.comm.upload_output)
print('url: %s' % client.get_webpage())
print('warnings: %s' % client.comm.warnings)
exporter = client.get_itol_export()
"""

exporter = ItolExport()
exporter.set_export_param_value('tree', '162245222240401575528925')

exporter.set_export_param_value('format', 'svg')
exporter.set_export_param_value('datasets_visible', ','.join([str(i) for i in range(dataset_count)]))
exporter.export(export_location)

Reading the output files, they contain the text Invalid SVG received from headless browser. This is an issue with itol's server. I sent an email to Ivica Letunic about it.

[iletunic]

The issue was caused by the timeout of phantomjs, since datasets are normally loaded one by one, which was taking too long... I added some changes to iTOL, so all dataset information will be bulk-loaded together with the initial tree data, which should fix the problem, and make exporting trees with several visible datasets much faster.

cheers

[albertyw]

Reran the above test case and it's exporting correctly now.

[jeffreyjeromequinn]

Hello Albert and Ivica,

Thanks so much for handling this. I can confirm, too, that the example dataset correctly exports now. Great!

Unfortunately, with considerably larger datasets (e.g. >10,000 leaves with ~20 visible datasets, I am still getting the same error. (Indeed, with >10,000 leaves and just one visible dataset, it also produces the error.) I understand that these are especially large trees and datasets. I've attached one such example below.

Thanks so much. Best wishes, Jeff

LG1_iTOL.zip