Reuse visualisation provided by bio.tools/TESS in biii

PerrineGilloteaux commented 4 years ago

for workflow visualisation

hmenager commented 4 years ago

a few pointers for workflow-like visualization tools:

a cytoscapejs plugin developped for TESS https://tess.elixir-europe.org/workflows/meta-pipe
a dedicated workflow viewer like in https://dev.workflowhub.eu/workflows/32?version=1

hmenager commented 4 years ago

a few other links:

the main cytoscape js repo: https://github.com/cytoscape/cytoscape.js
the panzoom plugin (also used in TeSS https://github.com/cytoscape/cytoscape.js-panzoom)

hmenager commented 4 years ago

after investigating the TeSS repo, here's my understanding of how it works:

the workflows are stored in the database,
the JSON is generated from this database and returned to the frontend,
this JSON (that includes the positions of the different nodes) is generated and handled by cytoscapejs for frontend display.

Note that the users can adjust and save the layout to improve readability of each of the nodes.

sample JSON:

{
    "nodes": [{
        "data": {
            "name": "Raw reads",
            "description": "",
            "html_description": "",
            "color": "#FFFFFF",
            "font_color": "rgb(0, 0, 0)",
            "associatedResources": [],
            "id": "724724b0-2dbb-4eb3-a387-f35b7625b03f"
        },
        "position": {
            "x": 509.2222222222223,
            "y": -30.222222222222225
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "SeqPrep",
            "description": "Paired-end overlapping reads are merged into a single longer read - we do not perform assembly.",
            "html_description": "\u003cp\u003ePaired-end overlapping reads are merged into a single longer read - we do not perform assembly.\u003c/p\u003e\n",
            "color": "#0000FF",
            "font_color": "rgb(255, 255, 255)",
            "associatedResources": [{
                "title": "SeqPrep",
                "url": "https://bio.tools/tool/SeqPrep/version/none",
                "type": "tools"
            }],
            "id": "1eea750d-b0bd-462d-89ea-0d71d0eb1b0c"
        },
        "position": {
            "x": 509.33333333333326,
            "y": 27.444444444444414
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "Initial reads",
            "description": "",
            "html_description": "",
            "color": "#FFFFFF",
            "font_color": "rgb(0, 0, 0)",
            "associatedResources": [],
            "id": "95034175-2699-4e86-a025-0630f3a6d61e"
        },
        "position": {
            "x": 510.8888888888889,
            "y": 83.11111111111111
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "QC",
            "description": "Low quality trimming (low quality ends and sequences with \u003e 10% undetermined nucleotides removed). Adapter sequences removed using Biopython SeqIO package.\nSequences \u003c 100 nucleotides in length removed.\n",
            "html_description": "\u003cp\u003eLow quality trimming (low quality ends and sequences with \u0026gt; 10% undetermined nucleotides removed). Adapter sequences removed using Biopython SeqIO package.\nSequences \u0026lt; 100 nucleotides in length removed.\u003c/p\u003e\n",
            "color": "#0000FF",
            "font_color": "rgb(255, 255, 255)",
            "associatedResources": [{
                "title": "Trimmomatic",
                "url": "https://bio.tools/tool/Trimmomatic/version/none",
                "type": "tools"
            }, {
                "title": "BioPython",
                "url": "https://bio.tools/tool/Biopython/version/none",
                "type": "tools"
            }],
            "id": "752bd870-3a30-408e-ac74-ea89b269f22e"
        },
        "position": {
            "x": 510.2222222222223,
            "y": 138.22222222222223
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "Processed reads",
            "description": "",
            "html_description": "",
            "color": "#FFFFFF",
            "font_color": "rgb(0, 0, 0)",
            "associatedResources": [],
            "id": "6438c65d-0e94-4cc9-8691-dec0e2f02c7b"
        },
        "position": {
            "x": 512.3333333333333,
            "y": 193.5555555555556
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "rRNA selector",
            "description": "Prokaryotic rRNA reads are filtered. We use the hidden Markov models to identify rRNA sequences.\n",
            "html_description": "\u003cp\u003eProkaryotic rRNA reads are filtered. We use the hidden Markov models to identify rRNA sequences.\u003c/p\u003e\n",
            "color": "#0000FF",
            "font_color": "rgb(255, 255, 255)",
            "associatedResources": [{
                "title": "rRNASelector",
                "url": "https://bio.tools/tool/rRNASelector/version/none",
                "type": "tools"
            }],
            "id": "ebed25e0-e765-4653-85d0-98bb4d6be6e4"
        },
        "position": {
            "x": 515.1111111111112,
            "y": 250.5555555555555
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "16S rRNA",
            "description": "",
            "html_description": "",
            "color": "#FFFFFF",
            "font_color": "rgb(0, 0, 0)",
            "parent": "a7578371-31a6-416c-a4db-dbd14864c236",
            "associatedResources": [],
            "id": "53d7fcae-b823-4afd-8483-ec5f46994836"
        },
        "position": {
            "x": 402.7777777777778,
            "y": 410.4444444444444
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": "visible"
    }, {
        "data": {
            "name": "FragGeneScan",
            "description": "Reads with predicted coding sequences (pCDS) above 60 nucleotides in length.",
            "html_description": "\u003cp\u003eReads with predicted coding sequences (pCDS) above 60 nucleotides in length.\u003c/p\u003e\n",
            "color": "#0000FF",
            "font_color": "rgb(255, 255, 255)",
            "parent": "2b6b4f77-f7c3-4d26-9460-42c60cd9fe1a",
            "associatedResources": [{
                "title": "FragGeneScan",
                "url": "https://bio.tools/tool/FragGeneScan/version/none",
                "type": "tools"
            }],
            "id": "db111896-c183-4af2-b17b-7fb66ea6d4ff"
        },
        "position": {
            "x": 633.888888888889,
            "y": 412.88888888888886
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "InterProScan",
            "description": "Matches are generated against predicted CDS, using a sub set of databases (Pfam, TIGRFAM, PRINTS, PROSITE patterns, Gene3d) from InterPro release 31.0. A summary of Gene Ontology (GO) terms derived from InterPro matches to your sample is provided. It is generated using a reduced list of GO terms called GO slim (version goslim_goa 2012).",
            "html_description": "\u003cp\u003eMatches are generated against predicted CDS, using a sub set of databases (Pfam, TIGRFAM, PRINTS, PROSITE patterns, Gene3d) from InterPro release 31.0. A summary of Gene Ontology (GO) terms derived from InterPro matches to your sample is provided. It is generated using a reduced list of GO terms called GO slim (version goslim_goa 2012).\u003c/p\u003e\n",
            "color": "#0000FF",
            "font_color": "rgb(255, 255, 255)",
            "parent": "2b6b4f77-f7c3-4d26-9460-42c60cd9fe1a",
            "associatedResources": [{
                "title": "InterProScan",
                "url": "https://bio.tools/tool/interproscan_5/version/1",
                "type": "tools"
            }],
            "id": "71ecf522-0350-4c04-b8a8-79271241e1d0"
        },
        "position": {
            "x": 635,
            "y": 522.8888888888888
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "QIIME",
            "description": "16s rRNA are annotated using the Greengenes reference database (default closed-reference OTU picking protocol with Greengenes 13.8 reference with reverse strand matching enabled).",
            "html_description": "\u003cp\u003e16s rRNA are annotated using the Greengenes reference database (default closed-reference OTU picking protocol with Greengenes 13.8 reference with reverse strand matching enabled).\u003c/p\u003e\n",
            "color": "#0000FF",
            "font_color": "rgb(255, 255, 255)",
            "parent": "a7578371-31a6-416c-a4db-dbd14864c236",
            "associatedResources": [{
                "title": "QIIME",
                "url": "https://bio.tools/tool/QIIME/version/1",
                "type": "tools"
            }],
            "id": "0c66c0f1-e017-449b-b2e8-e58e3a659a88"
        },
        "position": {
            "x": 402.2222222222223,
            "y": 476.5555555555555
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": "visible"
    }, {
        "data": {
            "name": "Functional analysis",
            "description": "",
            "html_description": "",
            "color": "#EEE8AA",
            "font_color": "rgb(0, 0, 0)",
            "associatedResources": [],
            "id": "2b6b4f77-f7c3-4d26-9460-42c60cd9fe1a"
        },
        "position": {
            "x": 634.4444444444446,
            "y": 439.2777777777777
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "Taxonomic analysis",
            "description": "",
            "html_description": "",
            "color": "#EEE8AA",
            "font_color": "rgb(0, 0, 0)",
            "associatedResources": [],
            "id": "a7578371-31a6-416c-a4db-dbd14864c236"
        },
        "position": {
            "x": 402.50000000000006,
            "y": 413.7777777777777
        },
        "group": "nodes",
        "removed": false,
        "selected": true,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "reads with rRNA",
            "description": "",
            "html_description": "",
            "color": "#FFFFFF",
            "font_color": "rgb(0, 0, 0)",
            "parent": "a7578371-31a6-416c-a4db-dbd14864c236",
            "associatedResources": [],
            "id": "152646a7-e6b3-4c22-aa1d-17080bc8ba4a"
        },
        "position": {
            "x": 402.7777777777778,
            "y": 350.99999999999994
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": "visible"
    }, {
        "data": {
            "name": "reads with rRNA masked",
            "description": "",
            "html_description": "",
            "color": "#FFFFFF",
            "font_color": "rgb(0, 0, 0)",
            "parent": "2b6b4f77-f7c3-4d26-9460-42c60cd9fe1a",
            "associatedResources": [],
            "id": "54d52ca9-7af4-4751-8684-64b944a9dd8f"
        },
        "position": {
            "x": 634.4444444444445,
            "y": 355.6666666666666
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "name": "Predicted CDS",
            "description": "",
            "html_description": "",
            "color": "#FFFFFF",
            "font_color": "rgb(0, 0, 0)",
            "parent": "2b6b4f77-f7c3-4d26-9460-42c60cd9fe1a",
            "associatedResources": [],
            "id": "81860ebe-901d-4a4f-b6d5-2d1a8b5b8ef9"
        },
        "position": {
            "x": 634.4444444444446,
            "y": 472.33333333333326
        },
        "group": "nodes",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }],
    "edges": [{
        "data": {
            "source": "152646a7-e6b3-4c22-aa1d-17080bc8ba4a",
            "target": "53d7fcae-b823-4afd-8483-ec5f46994836",
            "id": "1ccc59b8-e226-424f-9d3c-674eec02aa0e"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "53d7fcae-b823-4afd-8483-ec5f46994836",
            "target": "0c66c0f1-e017-449b-b2e8-e58e3a659a88",
            "id": "7b59f583-d249-4b0e-a6ae-c659bd7bb86b"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "ebed25e0-e765-4653-85d0-98bb4d6be6e4",
            "target": "a7578371-31a6-416c-a4db-dbd14864c236",
            "id": "ef7a3fac-3bba-4dc6-ac21-0a861b0bc168"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "724724b0-2dbb-4eb3-a387-f35b7625b03f",
            "target": "1eea750d-b0bd-462d-89ea-0d71d0eb1b0c",
            "id": "df9ed1ff-0e87-426a-a800-4eaf51af2887"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "1eea750d-b0bd-462d-89ea-0d71d0eb1b0c",
            "target": "95034175-2699-4e86-a025-0630f3a6d61e",
            "id": "710a874e-f967-4143-9f69-638f3ac65e22"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "95034175-2699-4e86-a025-0630f3a6d61e",
            "target": "752bd870-3a30-408e-ac74-ea89b269f22e",
            "id": "39547dd1-b9d4-4090-bc9a-1195626dd9f6"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "752bd870-3a30-408e-ac74-ea89b269f22e",
            "target": "6438c65d-0e94-4cc9-8691-dec0e2f02c7b",
            "id": "7ed4f4ae-7a3b-4ae7-a0ce-f3c0bc489c56"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "6438c65d-0e94-4cc9-8691-dec0e2f02c7b",
            "target": "ebed25e0-e765-4653-85d0-98bb4d6be6e4",
            "id": "ce68293e-cf85-477c-9149-c1094ae42935"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "ebed25e0-e765-4653-85d0-98bb4d6be6e4",
            "target": "2b6b4f77-f7c3-4d26-9460-42c60cd9fe1a",
            "id": "87a7636a-f4b9-4771-b325-638ee781ca41"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": ""
    }, {
        "data": {
            "source": "54d52ca9-7af4-4751-8684-64b944a9dd8f",
            "target": "db111896-c183-4af2-b17b-7fb66ea6d4ff",
            "id": "3b7121d5-c253-4a2f-947c-f7d00907f02d"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": "hidden"
    }, {
        "data": {
            "source": "db111896-c183-4af2-b17b-7fb66ea6d4ff",
            "target": "81860ebe-901d-4a4f-b6d5-2d1a8b5b8ef9",
            "id": "533e3f57-7f19-4f24-880d-f18bae9c0b9a"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": "hidden"
    }, {
        "data": {
            "source": "81860ebe-901d-4a4f-b6d5-2d1a8b5b8ef9",
            "target": "71ecf522-0350-4c04-b8a8-79271241e1d0",
            "id": "99fe5c9f-7deb-4fc1-a660-58c8eae42421"
        },
        "position": {},
        "group": "edges",
        "removed": false,
        "selected": false,
        "selectable": true,
        "locked": false,
        "grabbable": true,
        "classes": "hidden"
    }]
}

PerrineGilloteaux commented 4 years ago

Link to JSON export of workflows steps: http://test.biii.eu/wfsteps?_format=json

hmenager commented 4 years ago

link to TeSS OpenAPI workflow retrieval web service: https://app.swaggerhub.com/apis/fbacall/TeSS-JSON-API/0.2.0#/workflows/get_workflows__slug_

hmenager commented 4 years ago

The link to the "more reusable" javascript code (kindly provided by @inkuzmin is https://gitlab.com/njall/workflow-js. instructions to set up and run are straightforward ;)

albangaignard commented 4 years ago

Here is an overview of dynamic graph visualization of tools participating in Biii worklfows.

Full JavaScript (Cytoscape.js based) so it can be integrated into the Drupal web site. Drupal integration to be done.

albangaignard commented 4 years ago

@hmenager, @PerrineGilloteaux, @alat54 the JS code is availabe here: https://github.com/NeuBIAS/bise-core-ontology/tree/master/wf-html-vis.

Right now, it can be integrated yet into a Drupal page.

albangaignard commented 4 years ago

@PerrineGilloteaux, can you add the name of the step in the workflow drupal view ?

hmenager commented 4 years ago

to write down what we just discussed, what you will get from the cytoscapejs API has to be programmed. In particular, the size of the diagram has first to be decided, and then the code adapted to reflect the "rules". The cytoscapejs documentation can be very helpful with that respect: https://js.cytoscape.org/

PerrineGilloteaux commented 4 years ago

@alat54 Could you point out where you did incorporate the javascript from Alban and which view was modified? To check that the test.biii.eu version can support it? https://github.com/NeuBIAS/bise-core-ontology/tree/master/wf-html-vis.

PerrineGilloteaux commented 4 years ago

It does work, for example , http://test.biii.eu/node/1622 but not correctly for http://test.biii.eu/node/50 : = http://bisescratch.lndo.site (or test.biii or biii)/performing-automatic-registration-clem?_format=json -> actually from the json you only get the list of steps (paragraph) that you read then from http://bisescratch.lndo.site/wfsteps?_format=json

PerrineGilloteaux commented 4 years ago

So what was missing was a supplmenytary exposition so you can differntiate between the two steps spot finder in the example above (/node/50). I am moiving the label as an bug/enhancement , not related to the staus in Bordeaux (we let it like that at this stage)

PerrineGilloteaux commented 4 years ago

SO: it is not possible for you @albangaignard to differentiate in this actual state: (Locally bisescratch !)I need to edit the represnetation of workflow itself: replacing previous step by a paragraph of type workflow step. I post here once done.

PerrineGilloteaux commented 4 years ago

And at the same time add some label on the step representation

PerrineGilloteaux commented 4 years ago

We need this to be push to production in the next round even if not percet because we need more wf to be curated to make it better @miura

miura commented 4 years ago

@albangaignard @PerrineGilloteaux

For the development purpose, the best is to test adding a new workflow node in your local just for the purpose of development of these new functionalities, and do local rounds of testing/adjusting configurations, codes and contents.

The main reason is that if you try to modify the database in the production server, it takes too much time and effort.

the change in the biii.eu database needs to be exported as a SQL file, 2. copy that file to local, 3. import that database to the local and then 4. test different configurations, export the configuration, in local 5. pull request changes to the test server, wait for reviewing ... n. change in the configuration tests in the production site, n+1 change the workflow, and so on. It takes too much effort if you involve the deployed website directly for development.

So I recommend doing development locally using a test workflow node, and when it becomes successful and satisfactory, push and pull request.

PerrineGilloteaux commented 1 year ago

to reopen once workflow visu is defined for further improvements since a first version is now already deployed.

NEUBIAS / bise

Reuse visualisation provided by bio.tools/TESS in biii #129