Import Cohort by URL

[jduke99]

Desired Behavior

On Cohort page, be able to enter a URL of another Atlas instance cohort (e.g., http://www.ohdsi.org/web/atlas/#/cohortdefinition/922826) and have it automatically import without all the copying and pasting.

Actual behavior

Need to click around and copy and paste

Steps to reproduce behavior

N/A

[fdefalco]

I might suggest that instead of pointing at the ATLAS URL and have to deal with scraping the interface that we instead make the cohort expression WebAPI URL visible on the cohort definition page and then allow an import using that URL instead.

[t-abdul-basser]

@fdefalco I agree.

[fdefalco]

@jduke99 - How about a button on the toolbar to access the link:

Followed by a modal window with the link in a text box that you can copy/paste?

Still need to work out the import...

[jduke99]

++ @fdefalco's "Link to Cohort Definition" recommendation. Exactly what I was thinking.

[schuemie]

A related request: since this option (using a URL) only works if the URL is public (e.g. I can't use it to copy a cohort definition from the private JnJ Atlas to the public OHDSI Atlas), could we make the copy-pasting bit more intuitive?

Right now, to copy a cohort I need to

• On the source Atlas, open the cohort, go to Export, go to JSON, select all text, and press ctrl+C
• On the target Atlas, create a new cohort, go to Export, go to JSON, press ctrl+V, and click reload

This seems rather arbitrary (I need to go to Export to import? Why specifically the JSON representation?), and is not known to many. Having a 'Copy to clipboard' and 'Paste from c;lipboard' button, or at least a 'Import' tab next to the 'Export' tab might help.

On a similar note, the process for copy-pasting concept sets is very different from copy-pasting cohorts. Whatever we decide for cohorts, could be do the same for concept sets?

Same for estimation studies? Prediction studies?

[gklebanov]

@schuemie

"could we make the copy-pasting bit more intuitive?", "or at least a 'Import' tab next to the 'Export' tab might help."

Agree! Or, a simple and quick solution - rename "Export" tab to be "Export / Import", as well as adding simple on page instructions

@jduke99 @schuemie @pavgra "On Cohort page, be able to enter a URL of another Atlas instance cohort (e.g., http://www.ohdsi.org/web/atlas/#/cohortdefinition/922826) and have it automatically import without all the copying and pasting." "since this option (using a URL) only works if the URL is public (e.g. I can't use it to copy a cohort definition from the private JnJ Atlas to the public OHDSI Atlas), could we make the copy-pasting bit more intuitive?"

Agree with both on this. ATLAS is typically installed behind a firewall and thus it makes sharing cohorts via direct link impossible in most cases. In fact, the "Cohort import from ATLAS" functionality have been in implemented in ARACHNE exactly as described above e.g. it reads the JSON definition out of ATLAS (then we generate SQL based on that). But ARACHNE Central and Data Node communicate asynchronously via messaging which makes communication via firewall possible.

For example: ATLAS - http://www.ohdsi.org/web/atlas/#/cohortdefinition/922826 Link for cohort defintion in JSON - http://api.ohdsi.org/WebAPI/cohortdefinition/922826

Then, in ARACHNE we use CirceBE to translate the JSON into OHDSI SQL. Then it uses SqlTranslate and SqlRender - to convert to DB-specific SQL

here is this code in ARACHNE for your reference https://github.com/OHDSI/ArachneNodeAPI/blob/7cfa2ef96b6dd64a5aa86cdd4d4cd27c893f24c5/src/main/java/com/odysseusinc/arachne/datanode/service/client/atlas/AtlasClient.java

On a similar note, the process for copy-pasting concept sets is very different from copy-pasting cohorts. Whatever we decide for cohorts, could be do the same for concept sets?

Absolutely, great idea. Same goes for Import/Export where I believe we should define and use the same consistent JSON, including additional metadata attributes that would make sharing easier. Again, we do import all of the ATLAS / WebAPI artifacts via JSON in the similar fashion.

In fact, ARACHNE was created for the purpose of conducting cross-site federated studies and sharing code between various firewall protected sites was one of the requirements.

[jduke99]

@gklebanov This is great to know that ARACHNE has all this functionality. We definitely want to connect all the JSON quickly for multi-site studies. In my particular case, I just needed to import a bunch of cohort definitions from another library to my library. But no doubt we have all the guts already in place.

How to proceed on this?

[gklebanov]

@jduke99 hey Jon. Fantastic requirements - and I think the keywords here, as you stated it - "needed to import a bunch of cohort definitions from another library to my library".

It sounds like (and I agree) it is a bit time consuming for you to do one cohort at at time - as it works today. So, in addition to the cosmetical changes of having "Export / Import" screen polished, maybe the requirement should be formulated as being able to do "bulk export " and "bulk import"? I would imagine a screen in ATLAS where users can select cohorts (check boxes) and then click export button which would spit out a file with a number of JSON definitions inside. On a receiving side, have "Bulk Import" button that allows users to select that file and get cohorts re-created in bulk.

By the way, for this to work and to real fix our import/export - we need to extend an existing cohort JSON to include critical metadata attributes, including cohort name (mandatory), author (optional), GUID (as we discussed in another forum - should be mandatory) etc...

Again, creating import/export via API web service calls is a viable approach (and possibly complimentary) but you will find that in many cases another WebAPI is behind a firewall so it would not work too well.

If we agree on our vision and requirements here, I would be happy for Odysseus to contribute to these features.

[cgreich]

@fdefalco:

While you are at it: Create an import, and in the same place folks can either paste (like now), or open a file (maybe many), or paste a URL. And it will also pre-populate the same name.

@gklebanov:

Make sure that the Arachne UI follows that of ATLAS, so people don't get confused about the possibility to do the same thing in different ways. Or even call ATLAS. The more we avoid duplicating similar functionality, the better.

[anthonysena]

Closing this issue as the original ask for a way to get the URL to the WebAPI endpoint for the cohort definition was satisfied. The usability of ATLAS will be addressed in v3.0 and some of the ideas expressed here seem to fit in with #272 so I'm linking to that issue from here.