Graphics details upon export

nilsoberg commented 2 years ago

The latest Cytoscape GUI 3.10.0 snapshots include an option for turning off graphics details upon image export. The latest py4cytoscape code includes a toggle_graphics_details function. When I call this before exporting an image using the lastest py4cytoscape code in combination with Cytoscape 3.10.0 snapshot, there isn't any difference. Could I ask if there is a way to export images without applying styling? The image export runs MUCH faster on the snapshot without graphics details during export.

bdemchak commented 2 years ago

Hi, Nils ... wonderful question.

As background, the existing py4cytoscape call (https://py4cytoscape.readthedocs.io/en/latest/reference/generated/py4cytoscape.network_views.toggle_graphics_details.html?highlight=level%20of%20detail#py4cytoscape-network-views-toggle-graphics-details) invokes the long-standing LOD (Level of Detail) operation ... more than you'd ever want to know is here: https://manual.cytoscape.org/en/3.9.1/Rendering_Engine.html?highlight=level%20of%20detail#what-is-level-of-detail-lod

I think you're referring to the new 3.10 export, which I have heard raves about. I'll look into it and get back to you.

bdemchak commented 2 years ago

Hi, Nils ... we're still working on this. There is a new parameter for View Export, and its name is allGraphicsDetails= ... but it's not working when accessed from Cytoscape Automation. It'll likely be at least a week before we know more.

nilsoberg commented 2 years ago

I will add that parameter into my workflow in anticipation of the Automation update. Thank you for your help!

bdemchak commented 2 years ago

Hi, Nils --

One of the engineers offers some advice as a workaround until Cytoscape itself can be fixed:

The values in the image export dialog are saved to a props file in the CytoscapeConfiguration folder. If you export a network image with "Show All Graphics Details" unchecked, then close Cytoscape, the value of the checkbox is saved. If you then restart cytoscape and run the 'view export' command it will be like if you did allGraphicsDetails=false

You can see this by going to the CytoscapeConfiguration folder, find the file named writer.org.cytoscape.io.internal.write.graphics.PNGWriter.props, there should be a line allGraphicsDetails=false.

This actually isn't a good thing because changing something in the UI shouldn't have the side effect of changing the behaviour of a command. But it might be a good enough workaround for now.

nilsoberg commented 2 years ago

This will work for the time being. I have a singularity container that's starts up a brand new instance of Cytoscape within the container, exports an image, then is closed and the singularity instance is closed. I manually create a props file when building the image so that I can specify the REST API version without having to open the GUI and then close it. I will add this flag to the props file when I build the image.

Thanks so much! I think this should solve our problem until a production version is released. Your responsiveness to my queries is always much appreciated.

Nils

From: Barry Demchak @.> Sent: Tuesday, April 26, 2022, 6:10 PM To: cytoscape/py4cytoscape @.> Cc: Oberg, Nils O @.>; Author @.> Subject: Re: [cytoscape/py4cytoscape] Graphics details upon export (Issue #81)

Hi, Nils --

One of the engineers offers some advice as a workaround until Cytoscape itself can be fixed:

The values in the image export dialog are saved to a props file in the CytoscapeConfiguration folder. If you export a network image with "Show All Graphics Details" unchecked, then close Cytoscape, the value of the checkbox is saved. If you then restart cytoscape and run the 'view export' command it will be like if you did allGraphicsDetails=false

You can see this by going to the CytoscapeConfiguration folder, find the file named writer.org.cytoscape.io.internal.write.graphics.PNGWriter.props, there should be a line allGraphicsDetails=false.

This actually isn't a good thing because changing something in the UI shouldn't have the side effect of changing the behaviour of a command. But it might be a good enough workaround for now.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/cytoscape/py4cytoscape/issues/81*issuecomment-1110333017__;Iw!!DZ3fjg!96XRqVg5j8P4Ki4JXWK6WQL1EbxyjcCPTHsOBTYHIlE5ZluLWOUrmmr2-zFfPfDgil0mYtm0T60Wn6ajipNiukhbpw$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AI3P2JLAGBCGAOMJUWXAKDLVHBZWHANCNFSM5UC5ETNA__;!!DZ3fjg!96XRqVg5j8P4Ki4JXWK6WQL1EbxyjcCPTHsOBTYHIlE5ZluLWOUrmmr2-zFfPfDgil0mYtm0T60Wn6ajipPTSCHwsg$. You are receiving this because you authored the thread.Message ID: @.***>

bdemchak commented 2 years ago

Hi, Nils ... very much appreciate your kind words!

Would you be interested in telling us more about your Singularity image work? This sounds very broadly useful!

nilsoberg commented 2 years ago

Hi Barry.

The idea behind our code is to start up an instance of Cytsocape using a singularity container, interact with it using the CyREST API using a separate singularity container to export an image of a network.

We are exporting images for tens of thousands (and soon even hundreds of thousands) sequence similarity networks (XGMML files). We do this on a cluster. We’ve currently been using the GUI version of Cytoscape via Xvfb in a job script, then running a singularity image with custom code that wraps py4cytoscape to interact with the instance of Cytoscape to generate the image. This process has a couple of issues:

You have to run Cytoscape GUI once to set the props file so that the REST API version is saved
Multiple dependencies need to be installed, i.e. Java, Xvfb, Python, etc, and this is not generally applicable (I work with our cluster team to install software as necessary, but would like to avoid having to ask them every time I need something)
We have a 15-20% failure rate where Cytoscape just closes without generating an image, and no message appears in the output
Sometimes images get exported with a large amount of whitespace around the network
Since there are multiple versions of Cytoscape running at any given time in order to batch process the networks, conflicts can occur in the karaf cache data (I am not sure if that’s the proper description), so we have to copy the karaf data from the base CytoscapeConfiguration directory to a temp directory and tell karaf to use that data directory. This happens every time we start up a Cytoscape instance, and then also we have to clean it up.

The idea of wrapping all of this into a container is attractive because it would allow us to easily support various versions of Cytoscape, different memory configs, cleanup code, include various utilties, etc., making running Cytoscape for exporting images tidier. Singularity is the only thing required to run. When building the singularity image, the properties are automatically set (i.e. REST API version, and the flag for image export that we have been discussing). You still have to bind an external directory to the image on run-time in order for Cytoscape to write the cache files to, but this is still cleaner than the approach we’ve been taking.

We also have a py4cytoscape image that interacts with Cytoscape (whether started with the above process or image) to generate the image. This one is very useful because we can wrap all of the dependencies inside of the image without having to worry about Python dependency installation and such.

Here is the repo that contains the code and image definitions if you’re interested:

https://github.com/EnzymeFunctionInitiative/cytoscape-util/tree/dev-local-py4cytoscape

We are currently using a copy of a recent version of the py4cytoscape package. At some point I want to automate the download of the latest version in the repo, because the version of py4cytoscape when doing a pip install is way out of date.

It’s still a work in progress. However, we have successfully done this (with some manual exports) for this site:

https://radicalsam.org/

Go here for an example:

https://radicalsam.org/explore.php?id=cluster-1-1-1&v=3.0&as=110

Navigate through by clicking the Next button to see the various exports.

We have to generate images for some of the larger networks by hand because Cytoscape sometimes provides an image that has a layout that results in a single point. Doing this through the GUI results in better results.

We are getting ready to refresh the site, hoping to do it twice a year, anticipating > 100,000 networks/images. We are also planning on creating sites like the aforementioned one for other protein superfamilies.

We are really looking forward to 3.10, with the hope that this solves some of the issues.

Nils

From: Barry Demchak @.> Sent: Wednesday, April 27, 2022 10:08 AM To: cytoscape/py4cytoscape @.> Cc: Oberg, Nils O @.>; Author @.> Subject: Re: [cytoscape/py4cytoscape] Graphics details upon export (Issue #81)

Hi, Nils ... very much appreciate your kind words!

Would you be interested in telling us more about your Singularity image work? This sounds very broadly useful!

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/cytoscape/py4cytoscape/issues/81*issuecomment-1111118588__;Iw!!DZ3fjg!611x8f5V5_EyXk-XarB1UvoWqwM6EXsCUYBKlUJsJEWh_vPVisDnBlliIR3xq4n4Pz3-lWBz4g1TQPF7f_1gMdrtzA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AI3P2JMLXHN4AF6HQR3L6CDVHFJ4DANCNFSM5UC5ETNA__;!!DZ3fjg!611x8f5V5_EyXk-XarB1UvoWqwM6EXsCUYBKlUJsJEWh_vPVisDnBlliIR3xq4n4Pz3-lWBz4g1TQPF7f_2lRPhExQ$. You are receiving this because you authored the thread.Message ID: @.***>

bdemchak commented 2 years ago

Hi, Nils ....

Thanks for all of the great detail!

We discussed this at our weekly meeting today ... lots to tell.

It sounds like one or more of our team members are in contact with either your PI or a team leader on this issue, too. Apparently, they recommended using Solidarity, and I can certainly agree. Good news there.

Generally, the developer team agrees that helping you get to a successful point is very important ... certainly by 3.10 if not much sooner. It's entirely likely that nightly Cytoscape builds will get us closer and closer to good solutions. So, we'll keep you in the loop, and good feedback would be very helpful.

I think the dependencies issue has also been encountered by others while trying to do a Docker version. So far, multiple groups have limited success ... e.g., Kozo Nashida in Japan, the Toronto Cytoscape group, and the San Francisco Cytoscape group. At some point, it would be good to bring heads together ... maybe soon.

The 15-20% failure rate could well be because of multiple Cytoscape instances using the same CytoscapeConfiguration directory. The developers think that new features in Solidarity enable writeable file systems that are valid for the life of the Solidarity instance. That should solve this. There's a different concept in Docker ... an overlay applied to a Docker instance that accomplishes the same thing. This will be an important discussion topic.

The developers are also working on a straight forward way to get rid of the initial REST API dialog box. It really messes up Docker/Solidarity.

Regarding the whitespace around a network, is it possible that the network includes an extreme outlier, and that fitting the network shrinks the main network so the outlier can fit in, too? Alternately, could it be that a zoom factor has been applied, but the network hasn't be re-laid out?

It seems like all of the above would address all of the concerns you indicated. Is that so? Good to get feedback on this.

FYI, I'll be releasing an updated py4cytoscape onto PyPI in the next week or two ... but after that, py4cytoscape development will track 3.10 changes so you can take advantage of them. For example, we discovered a better way to accommodate the fast Export you're trying for. We'll make it available as soon as the 3.10 nightly build supports it.

Comments?

nilsoberg commented 2 years ago

Sorry for the delay in replying.

We are very grateful for our collaboration and your assistance. Thank you!

I haven’t tried Solidarity yet. It’s on my list.

I am happy to participate in discussions that you feel I would be able to contribute to.

In order to avoid the failures, I have made a complicated workaround that does something like this:

Create a new Cytoscape “install” by symlinking some directories and copying some other files into a unique directory.
Copy a vanilla CytoscapeConfiguration folder to the same directory. The vanilla folder has the proper cyrest.version set in cytoscape3.props so that the dialog isn’t required.
Modify the cytoscape.sh file in the new “install” changing the cytoscape.home and user.home parameters Java parameters in KARAF_OPTS to use the “install” directory

This is all managed by a series of scripts that are executed automatically by cron.

The whitespace is caused by the way the clusters are laid out, and since I have a fixed image size the images sometimes have a lot of space around them. I wrote a script that crops it. May not be the best way, but it gets the job done.

I have tested my code using the Cytoscape 3.10 06-29 snapshot and the latest py4cytoscape, and I am still having image generation issues.

Here is some information that may be relevant:

I am running into a strange image zoom issue. Sometimes the network is fitted properly and everything looks fine. Other times the network isn’t fitted: it’s like the image is zoomed in very close and cropped to a rectangle , and the image size isn’t respected. This only happens on networks that don’t have the warning from previously. If I use the same code in py4cytoscape with Cytoscape 3.8, the images are generated correctly.
The export_image function in the py4cytoscape code has a zoom parameter. It doesn’t seem to be acknowledged by 3.10 using the py4cytoscape version that I downloaded in April or in the most recent version. Cytoscape 3.8, doesn’t seem to respect that parameter either, or at least there is no apparent effect when I change it.
For large networks, the following error is displayed in the Cytoscape console during layout. The image is generated correctly, without the zoom issues I have described, but isn’t colored (i.e. the node.fillColor attribute is not respected). This occurs on both 3.8 and 3.10.

10:05:22.912 WARN [qtp1829412927-616] Network larger than view threshold Network factory creation finished in 52238 msec.

Sometimes nodes in large clusters are stacked on top of each other when using the non-OpenCL version of prefuse force directed layout.

I am wondering if some of these issues are related directly to Cytoscape as opposed to py4cytoscape, so I am going to forward this information to S. M. as well.

In case you are interested, here is the py4cytoscape logging code:

2022-06-30 09:57:21,767 [DEBUG] py4...: Calling cytoscape_ping('http://127.0.0.1:8001/v1') 2022-06-30 09:57:21,783 [DEBUG] py4...: ǀCalling cytoscape_version_info(base_url='http://127.0.0.1:8001/v1') 2022-06-30 09:57:21,783 [DEBUG] py4...: ǀǀCalling cyrest_get('version', base_url='http://127.0.0.1:8001/v1') 2022-06-30 09:57:21,783 [DEBUG] py4...: ǀǀHTTP GET(http://127.0.0.1:8001/v1/version) 2022-06-30 09:57:21,963 [DEBUG] py4...: ǀǀOK[200], content: {"apiVersion":"v1","cytoscapeVersion":"3.10.0-SNAPSHOT"} 2022-06-30 09:57:21,963 [DEBUG] py4...: ǀǀReturning 'cyrest_get': {'apiVersion': 'v1', 'cytoscapeVersion': '3.10.0-SNAPSHOT'} 2022-06-30 09:57:21,964 [DEBUG] py4...: ǀReturning 'cytoscape_version_info': {'apiVersion': 'v1', 'cytoscapeVersion': '3.10.0-SNAPSHOT', 'automat n': '1.5.0', 'py4cytoscapeVersion': '1.4.0'} 2022-06-30 09:57:21,965 [DEBUG] py4...: Returning 'cytoscape_ping': 'You are connected to Cytoscape!' 2022-06-30 09:57:21,965 [DEBUG] py4...: -------------------- 2022-06-30 09:57:21,965 [DEBUG] py4...: Calling import_network_from_file(base_url='http://127.0.0.1:8001/v1', file='/private_stores/gerlt/databas /rsam-4.0/cluster-3-2/dicing-50/ssn.xgmml') 2022-06-30 09:57:21,965 [DEBUG] py4...: ǀCalling commands_post('network load file file="/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluste -50/ssn.xgmml"', base_url='http://127.0.0.1:8001/v1') 2022-06-30 09:57:21,966 [DEBUG] py4...: ǀHTTP POST(http://127.0.0.1:8001/v1/commands/network/load%20file), json: {'file': '/private_stores/gerlt/ am/ip88/rsam-4.0/cluster-3-2/dicing-50/ssn.xgmml'} 2022-06-30 10:06:07,233 [DEBUG] py4...: ǀOK[200], content: { "data": { "networks":[ 122 ], "views":[]}, "errors":[] } 2022-06-30 10:06:07,234 [DEBUG] py4...: ǀReturning 'commands_post': {'networks': [122], 'views': []} 2022-06-30 10:06:11,238 [DEBUG] py4...: Returning 'import_network_from_file': {'networks': [122], 'views': []} 2022-06-30 10:06:11,238 [DEBUG] py4...: -------------------- 2022-06-30 10:06:11,239 [DEBUG] py4...: Calling get_network_suid(base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,239 [DEBUG] py4...: ǀCalling commands_post('network get attribute network="current" namespace="default" columnList="SUID"', b p://127.0.0.1:8001/v1') 2022-06-30 10:06:11,239 [DEBUG] py4...: ǀHTTP POST(http://127.0.0.1:8001/v1/commands/network/get%20attribute), json: {'network': 'current', 'name ault', 'columnList': 'SUID'} 2022-06-30 10:06:11,286 [DEBUG] py4...: ǀOK[200], content: { "data": [ { "SUID": 122 } ], "errors":[] } 2022-06-30 10:06:11,287 [DEBUG] py4...: ǀReturning 'commands_post': [{'SUID': 122}] 2022-06-30 10:06:11,287 [DEBUG] py4...: Returning 'get_network_suid': 122 2022-06-30 10:06:11,287 [DEBUG] py4...: -------------------- 2022-06-30 10:06:11,288 [DEBUG] py4...: Calling get_network_views(network=122, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,288 [DEBUG] py4...: ǀCalling get_network_suid(122, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,288 [DEBUG] py4...: ǀǀCalling cyrest_get('networks', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,288 [DEBUG] py4...: ǀǀHTTP GET(http://127.0.0.1:8001/v1/networks) 2022-06-30 10:06:11,303 [DEBUG] py4...: ǀǀOK[200], content: [122] 2022-06-30 10:06:11,303 [DEBUG] py4...: ǀǀReturning 'cyrest_get': [122] 2022-06-30 10:06:11,304 [DEBUG] py4...: ǀReturning 'get_network_suid': 122 2022-06-30 10:06:11,304 [DEBUG] py4...: ǀCalling cyrest_get('networks/122/views', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,304 [DEBUG] py4...: ǀHTTP GET(http://127.0.0.1:8001/v1/networks/122/views) 2022-06-30 10:06:11,319 [DEBUG] py4...: ǀNot Found[404], content: {"data":{},"errors":[{"status":404,"type":"urn:cytoscape:ci:cyrest-core:v1:netw rrors:6","message":"No Views Available for Network with SUID: 122","link":"file:/igbgroup/n-z/noberg/dev/ssn2image/test/cytoscape/temp/14/Cytosca ion/3/framework-cytoscape.log"}]} 2022-06-30 10:06:11,353 [DEBUG] py4...: ǀ'cyrest_get' exception CyError('In cyrest_get(): No Views Available for Network with SUID: 122') 2022-06-30 10:06:11,353 [DEBUG] py4...: Returning 'get_network_views': [] 2022-06-30 10:06:11,353 [DEBUG] py4...: -------------------- 2022-06-30 10:06:11,353 [DEBUG] py4...: Calling create_view(network=122, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,353 [DEBUG] py4...: ǀCalling get_network_suid(122, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,353 [DEBUG] py4...: ǀǀCalling cyrest_get('networks', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,353 [DEBUG] py4...: ǀǀHTTP GET(http://127.0.0.1:8001/v1/networks) 2022-06-30 10:06:11,365 [DEBUG] py4...: ǀǀOK[200], content: [122] 2022-06-30 10:06:11,365 [DEBUG] py4...: ǀǀReturning 'cyrest_get': [122] 2022-06-30 10:06:11,366 [DEBUG] py4...: ǀReturning 'get_network_suid': 122 2022-06-30 10:06:11,366 [DEBUG] py4...: ǀCalling get_network_view_suid(122, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,366 [DEBUG] py4...: ǀǀCalling cyrest_get('networks', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,366 [DEBUG] py4...: ǀǀHTTP GET(http://127.0.0.1:8001/v1/networks) 2022-06-30 10:06:11,379 [DEBUG] py4...: ǀǀOK[200], content: [122] 2022-06-30 10:06:11,379 [DEBUG] py4...: ǀǀReturning 'cyrest_get': [122] 2022-06-30 10:06:11,379 [DEBUG] py4...: ǀǀCalling get_network_views(122, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,379 [DEBUG] py4...: ǀǀǀCalling get_network_suid(122, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,379 [DEBUG] py4...: ǀǀǀǀCalling cyrest_get('networks', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,379 [DEBUG] py4...: ǀǀǀǀHTTP GET(http://127.0.0.1:8001/v1/networks) 2022-06-30 10:06:11,391 [DEBUG] py4...: ǀǀǀǀOK[200], content: [122] 2022-06-30 10:06:11,392 [DEBUG] py4...: ǀǀǀǀReturning 'cyrest_get': [122] 2022-06-30 10:06:11,392 [DEBUG] py4...: ǀǀǀReturning 'get_network_suid': 122 2022-06-30 10:06:11,392 [DEBUG] py4...: ǀǀǀCalling cyrest_get('networks/122/views', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,392 [DEBUG] py4...: ǀǀǀHTTP GET(http://127.0.0.1:8001/v1/networks/122/views) 2022-06-30 10:06:11,402 [DEBUG] py4...: ǀǀǀNot Found[404], content: {"data":{},"errors":[{"status":404,"type":"urn:cytoscape:ci:cyrest-core:v1:ne :errors:6","message":"No Views Available for Network with SUID: 122","link":"file:/igbgroup/n-z/noberg/dev/ssn2image/test/cytoscape/temp/14/Cytos ation/3/framework-cytoscape.log"}]} 2022-06-30 10:06:11,402 [DEBUG] py4...: ǀǀǀ'cyrest_get' exception CyError('In cyrest_get(): No Views Available for Network with SUID: 122') 2022-06-30 10:06:11,402 [DEBUG] py4...: ǀǀReturning 'get_network_views': [] 2022-06-30 10:06:11,403 [DEBUG] py4...: ǀReturning 'get_network_view_suid': None 2022-06-30 10:06:11,403 [DEBUG] py4...: ǀCalling commands_post('view create network="SUID:122" layout=True', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:06:11,403 [DEBUG] py4...: ǀHTTP POST(http://127.0.0.1:8001/v1/commands/view/create), json: {'network': 'SUID:122', 'layout': 'True' 2022-06-30 10:10:19,804 [DEBUG] py4...: ǀOK[200], content: { "data": [{"network":122,"view":31453274}], "errors":[] } 2022-06-30 10:10:19,804 [DEBUG] py4...: ǀReturning 'commands_post': [{'network': 122, 'view': 31453274}] 2022-06-30 10:10:19,804 [DEBUG] py4...: Returning 'create_view': 31453274 2022-06-30 10:10:19,805 [DEBUG] py4...: -------------------- 2022-06-30 10:10:19,805 [DEBUG] py4...: Calling layout_network(layout_name='force-directed', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:10:19,805 [DEBUG] py4...: ǀCalling get_network_suid(None, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:10:19,805 [DEBUG] py4...: ǀǀCalling commands_post('network get attribute network="current" namespace="default" columnList="SUID"', tp://127.0.0.1:8001/v1') 2022-06-30 10:10:19,805 [DEBUG] py4...: ǀǀHTTP POST(http://127.0.0.1:8001/v1/commands/network/get%20attribute), json: {'network': 'current', 'nam fault', 'columnList': 'SUID'} 2022-06-30 10:10:19,829 [DEBUG] py4...: ǀǀOK[200], content: { "data": [ { "SUID": 122 } ], "errors":[] } 2022-06-30 10:10:19,830 [DEBUG] py4...: ǀǀReturning 'commands_post': [{'SUID': 122}] 2022-06-30 10:10:19,830 [DEBUG] py4...: ǀReturning 'get_network_suid': 122 2022-06-30 10:10:19,830 [DEBUG] py4...: ǀCalling commands_post('layout force-directed network="SUID:122"', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:10:19,830 [DEBUG] py4...: ǀHTTP POST(http://127.0.0.1:8001/v1/commands/layout/force-directed), json: {'network': 'SUID:122'} 2022-06-30 10:12:55,223 [DEBUG] py4...: ǀOK[200], content: { "data": {}, "errors":[] } 2022-06-30 10:12:55,223 [DEBUG] py4...: ǀReturning 'commands_post': {} 2022-06-30 10:12:55,223 [DEBUG] py4...: Returning 'layout_network': {} 2022-06-30 10:12:55,224 [DEBUG] py4...: -------------------- 2022-06-30 10:12:55,224 [DEBUG] py4...: Calling fit_content(base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,224 [DEBUG] py4...: ǀCalling get_network_view_suid(None, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,224 [DEBUG] py4...: ǀǀCalling get_network_views(None, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,224 [DEBUG] py4...: ǀǀǀCalling get_network_suid(None, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,225 [DEBUG] py4...: ǀǀǀǀCalling commands_post('network get attribute network="current" namespace="default" columnList="SUID"' http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,225 [DEBUG] py4...: ǀǀǀǀHTTP POST(http://127.0.0.1:8001/v1/commands/network/get%20attribute), json: {'network': 'current', 'n default', 'columnList': 'SUID'} 2022-06-30 10:12:55,243 [DEBUG] py4...: ǀǀǀǀOK[200], content: { "data": [ { "SUID": 122 } ], "errors":[] } 2022-06-30 10:12:55,243 [DEBUG] py4...: ǀǀǀǀReturning 'commands_post': [{'SUID': 122}] 2022-06-30 10:12:55,243 [DEBUG] py4...: ǀǀǀReturning 'get_network_suid': 122 2022-06-30 10:12:55,243 [DEBUG] py4...: ǀǀǀCalling cyrest_get('networks/122/views', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,244 [DEBUG] py4...: ǀǀǀHTTP GET(http://127.0.0.1:8001/v1/networks/122/views) 2022-06-30 10:12:55,256 [DEBUG] py4...: ǀǀǀOK[200], content: [31453274] 2022-06-30 10:12:55,256 [DEBUG] py4...: ǀǀǀReturning 'cyrest_get': [31453274] 2022-06-30 10:12:55,256 [DEBUG] py4...: ǀǀReturning 'get_network_views': [31453274] 2022-06-30 10:12:55,257 [DEBUG] py4...: ǀReturning 'get_network_view_suid': 31453274 2022-06-30 10:12:55,257 [DEBUG] py4...: ǀCalling commands_post('view fit content view="SUID:31453274"', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,257 [DEBUG] py4...: ǀHTTP POST(http://127.0.0.1:8001/v1/commands/view/fit%20content), json: {'view': 'SUID:31453274'} 2022-06-30 10:12:55,294 [DEBUG] py4...: ǀOK[200], content: { "data": {}, "errors":[] } 2022-06-30 10:12:55,294 [DEBUG] py4...: ǀReturning 'commands_post': {} 2022-06-30 10:12:55,295 [DEBUG] py4...: Returning 'fit_content': {} 2022-06-30 10:12:55,295 [DEBUG] py4...: -------------------- 2022-06-30 10:12:55,297 [DEBUG] py4...: Calling toggle_graphics_details(base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,297 [DEBUG] py4...: ǀCalling cyrest_put('ui/lod', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,297 [DEBUG] py4...: ǀHTTP PUT(http://127.0.0.1:8001/v1/ui/lod) 2022-06-30 10:12:55,345 [DEBUG] py4...: ǀOK[200], content: {"message":"Toggled Graphics level of details."} 2022-06-30 10:12:55,345 [DEBUG] py4...: ǀReturning 'cyrest_put': {'message': 'Toggled Graphics level of details.'} 2022-06-30 10:12:55,345 [DEBUG] py4...: Returning 'toggle_graphics_details': {'message': 'Toggled Graphics level of details.'} 2022-06-30 10:12:55,345 [DEBUG] py4...: -------------------- 2022-06-30 10:12:55,346 [DEBUG] py4...: Calling export_image(filename='/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2/dicing-50/s png', type='PNG', units='pixels', height=1600, width=2000, zoom=0.0, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,346 [DEBUG] py4...: ǀCalling get_network_view_suid(None, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,346 [DEBUG] py4...: ǀǀCalling get_network_views(None, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,346 [DEBUG] py4...: ǀǀǀCalling get_network_suid(None, base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,347 [DEBUG] py4...: ǀǀǀǀCalling commands_post('network get attribute network="current" namespace="default" columnList="SUID"' http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,347 [DEBUG] py4...: ǀǀǀǀHTTP POST(http://127.0.0.1:8001/v1/commands/network/get%20attribute), json: {'network': 'current', 'n default', 'columnList': 'SUID'} 2022-06-30 10:12:55,362 [DEBUG] py4...: ǀǀǀǀOK[200], content: { "data": [ { "SUID": 122 } ], "errors":[] } 2022-06-30 10:12:55,363 [DEBUG] py4...: ǀǀǀǀReturning 'commands_post': [{'SUID': 122}] 2022-06-30 10:12:55,363 [DEBUG] py4...: ǀǀǀReturning 'get_network_suid': 122 2022-06-30 10:12:55,363 [DEBUG] py4...: ǀǀǀCalling cyrest_get('networks/122/views', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,363 [DEBUG] py4...: ǀǀǀHTTP GET(http://127.0.0.1:8001/v1/networks/122/views) 2022-06-30 10:12:55,375 [DEBUG] py4...: ǀǀǀOK[200], content: [31453274] 2022-06-30 10:12:55,375 [DEBUG] py4...: ǀǀǀReturning 'cyrest_get': [31453274] 2022-06-30 10:12:55,375 [DEBUG] py4...: ǀǀReturning 'get_network_views': [31453274] 2022-06-30 10:12:55,376 [DEBUG] py4...: ǀReturning 'get_network_view_suid': 31453274 2022-06-30 10:12:55,376 [DEBUG] py4...: ǀCalling sandbox_get_file_info('/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2/dicing-50/ .png', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,376 [DEBUG] py4...: ǀǀCalling commands_post('filetransfer getFileInfo fileName="/private_stores/gerlt/databases/rsam/ip88/rsa r-3-2/dicing-50/ssn_lg_fast2.png"', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,377 [DEBUG] py4...: ǀǀHTTP POST(http://127.0.0.1:8001/v1/commands/filetransfer/getFileInfo), json: {'fileName': '/private_sto tabases/rsam/ip88/rsam-4.0/cluster-3-2/dicing-50/ssn_lg_fast2.png'} 2022-06-30 10:12:55,396 [DEBUG] py4...: ǀǀOK[200], content: { "data": {"filePath":"/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2/dicing-50/ssn_lg_fast2.png","modifiedTime":"","isFile":false "errors":[] } 2022-06-30 10:12:55,396 [DEBUG] py4...: ǀǀReturning 'commands_post': {'filePath': '/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2 sn_lg_fast2.png', 'modifiedTime': '', 'isFile': False} 2022-06-30 10:12:55,396 [DEBUG] py4...: ǀReturning 'sandbox_get_file_info': {'filePath': '/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/clus ng-50/ssn_lg_fast2.png', 'modifiedTime': '', 'isFile': False} 2022-06-30 10:12:55,397 [DEBUG] py4...: ǀCalling commands_post('view export Units="pixels" Height="1600" Width="2000" OutputFile="/private_stores ases/rsam/ip88/rsam-4.0/cluster-3-2/dicing-50/ssn_lg_fast2.png" options="PNG" view="SUID:31453274"', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:12:55,397 [DEBUG] py4...: ǀHTTP POST(http://127.0.0.1:8001/v1/commands/view/export), json: {'Units': 'pixels', 'Height': '1600', 'W ', 'OutputFile': '/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2/dicing-50/ssn_lg_fast2.png', 'options': 'PNG', 'view': 'SUID:314 2022-06-30 10:14:17,230 [DEBUG] py4...: ǀOK[200], content: { "data": {"file":"/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2/dicing-50/ssn_lg_fast2.png"}, "errors":[] } 2022-06-30 10:14:17,230 [DEBUG] py4...: ǀReturning 'commands_post': {'file': '/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2/dici _fast2.png'} 2022-06-30 10:14:17,230 [DEBUG] py4...: Returning 'export_image': {'file': '/private_stores/gerlt/databases/rsam/ip88/rsam-4.0/cluster-3-2/dicing ast2.png'} 2022-06-30 10:14:17,231 [DEBUG] py4...: -------------------- 2022-06-30 10:14:17,231 [DEBUG] py4...: Calling command_quit(base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:14:17,231 [DEBUG] py4...: ǀCalling commands_post('command quit', base_url='http://127.0.0.1:8001/v1') 2022-06-30 10:14:17,231 [DEBUG] py4...: ǀHTTP POST(http://127.0.0.1:8001/v1/commands/command/quit), json: {} 2022-06-30 10:14:17,313 [DEBUG] py4...: ǀOK[200], content: { "data": {}, "errors":[] } 2022-06-30 10:14:17,313 [DEBUG] py4...: ǀReturning 'commands_post': {} 2022-06-30 10:14:17,314 [DEBUG] py4...: Returning 'command_quit': {} 2022-06-30 10:14:17,314 [DEBUG] py4...: --------------------

bdemchak commented 2 years ago

Hi, Nils --

Great to hear from you!

I looked into the export zoom not working ... I can duplicate that here. FYI, when I try to duplicate a py4cytoscape problem, I go to the Cytoscape Commands dialog and issue a command that py4cytoscape would generate. If Cytoscape chokes, we know the problem is in Cytoscape or in how I understand the command. For this zoom problem, I can see Cytoscape choking on the zoom parameter, so I have put the question to the Cytoscape developers.

As this thread is getting long, best to break into an Issue for each of your problems. That way, they can be tracked separately and all handled.

For each of your issues, would you mind creating a reproduce case and a separate Github Issue?

Thanks.

bdemchak commented 2 years ago

Hi, Nils --

More on the Export issue ... our group has put quite a bit of energy into understanding and solving this issue. We have some definite thoughts on this, and are ready to proceed.

Is there a particular test case you can give us? ... we'd like to see something that works in v3.8.2 and not in v10.

Thanks.

nilsoberg commented 2 years ago

I'm working on submitting separate issues with test cases, etc.

I am not able to reliably replicate the most problematic one (the image is rendered zoomed in to the middle of the network). This may somehow be related to running many instances at the same time, but I feel like that workflow is fairly foolproof. I am submitting that process and code example for how I am running multiple instances on the same system. However, this problem never occurs with 3.8.

An example of what is happening is attached. The first image is a properly-generated image. The second image is the zoom/crop problem image which I am still working on duplicating.

ssn_proper

ssn_zoom_crop

The workflow for making multiple instances of Cytoscape run simultaneously is as follows:

Create a $TEMP_DIR and create $TEMP_DIR/framework
Copy $CYTOSCAPE_APP_HOME/framework/[etc, data, instances] to $TEMP_DIR/framework
Symlink any remaining files in $CYTOSCAPE_APP_HOME/framework into $TEMP_DIR/framework
Copy $CYTOSCAPE_APP_HOME/cytoscape.sh to $TEMP_DIR (modify later)
Symlink any remaining files in $CYTOSCAPE_APP_HOME to $TEMP_DIR
Copy $HOME/CytoscapeConfiguration.clean_copy to $TEMP_DIR/CytoscapeConfiguration

Modify $TEMP_DIR/cytoscape.sh by replacing the export KARAF_OPTS line with the following:

USER_HOME=$HOME
# $TEMP_USER_HOME is an environment variable set by a master script that manages all of the running jobs
if [[ -n ${TEMP_USER_HOME+x} ]]; then
    export KARAF_OPTS=-Dcytoscape.home="$CYTOSCAPE_HOME_ABS"\ -Duser.home="$TEMP_USER_HOME"\ "$CYTOSCAPE_MAC_OPTS"
    USER_HOME=$TEMP_USER_HOME
else
    # Normal case
    export KARAF_OPTS=-Dcytoscape.home="$CYTOSCAPE_HOME_ABS"\ "$CYTOSCAPE_MAC_OPTS"
fi

Replace export KARAF_DATA with:

# Point to the temporary directory that was created
export KARAF_DATA=\"\${USER_HOME}/CytoscapeConfiguration/3/karaf_data\"

Create a script that does the following:

port=###
CYTOSCAPE=$TEMP_DIR/cytoscape.sh
export CHECK_ROOT_INSTANCE_RUNNING=false
export TEMP_USER_HOME=$TEMP_DIR # from above
xvfb-run --server-args="-screen 0 4000x3000x24" -d $CYTOSCAPE -R $port < /dev/zero

This is then submitted to a cluster, with a unique port. Additional scripts using the CyREST API are then submitted to the cluster with a matching port and node, to generate PNG from given networks.

This process runs many instances (up to 50) simultaneously, mostly working. But sometimes the zoom/crop issue appears with 3.10 . I can use the exact same code for 3.8 and I never have the zoom/crop problem.

nilsoberg commented 1 year ago

Hi Barry,

I just realized I didn’t respond to this email. My apologies.

We are very grateful for our collaboration and your assistance. Thank you!

I haven’t tried Solidarity yet. It’s on my list.

I am happy to participate in discussions that you feel I would be able to contribute to.

In order to avoid the failures, I have made a complicated workaround that does something like this:

Create a new Cytoscape “install” by symlinking some directories and copying some other files into a unique directory.
Copy a vanilla CytoscapeConfiguration folder to the same directory. The vanilla folder has the proper cyrest.version set in cytoscape3.props so that the dialog isn’t required.
Modify the cytoscape.sh file in the new “install” changing the cytoscape.home and user.home parameters Java parameters in KARAF_OPTS to use the “install” directory

This is all managed by a series of scripts that are executed automatically by cron.

The whitespace is caused by the way the clusters are laid out, and since I have a fixed image size the images sometimes have a lot of space around them. I wrote a script that crops it. May not be the best way, but it gets the job done.

A couple of things:

We are using the 3.10 snapshots to make images and are running into a few issues:

The export_image function in the py4cytoscape code has a zoom parameter. It doesn’t seem to be acknowledged by 3.10 using the py4cytoscape version that I downloaded in April. Images for large networks are generated correctly, but smaller networks are generated with the specified/requested image size, but the image is zoomed in by a large magnitude and the image is cropped to a box in the center.
- Cytoscape 3.8, doesn’t seem to respect that parameter either. There is no apparent effect when I change it. However, the images are fit correctly.
- When I switched to the latest version of py4cytocsape there is no layout, just a single dot. The API calls are given below.
We include a node attribute in our xgmml files for coloring the node (node.fillColor). Images exported for large networks in both 3.8 and 3.10 are generated without coloring, like Cytoscape isn’t respecting that node attribute, but networks that are smaller are colored. (Cytoscape problem vs py4cytoscape?)
For large clusters, the Prefuse Force-Directed Layout doesn’t actually perform a layout, all that is generated is a single node (or possibly all nodes stacked on top of each other). If the OpenCL version is used instead, the cluster layout is performed correctly. This happens in both versions of Cytoscape. (Cytoscape problem vs py4cytoscape?)

Could I ask if you have any advice for us on how to solve this problem?

Thanks in advance,

Nils

API call sequence:

cytoscape_ping (until alive) import_network_from_file get_network_views create_view (if not created) layout_network(layout_name="force-directed") fit_content toggle_graphics_details export_image(filename, PNG, pixels, height, width, zoom) command_quit

From: Barry Demchak @.> Sent: Thursday, April 28, 2022 4:58 PM To: cytoscape/py4cytoscape @.> Cc: Oberg, Nils O @.>; Author @.> Subject: Re: [cytoscape/py4cytoscape] Graphics details upon export (Issue #81)