[PLAN] Data

[YohannParis]

Explorer

• [ ] integrate tables and figures search (Ahmed: https://github.com/DARPA-ASKEM/TERArium/issues/382)
• [ ] integrate url extractions (Ahmed: https://github.com/DARPA-ASKEM/TERArium/issues/357)
• [ ] integrate citation network - design first
• [ ] search titles and abstract by default (Ahmed: https://github.com/DARPA-ASKEM/TERArium/issues/355)
• [ ] default page (jumping point, big search box, types of search, would reduce load time) - design
• [ ] results (better metadata + image) - design
• [ ] matrix: (clustering, use RCM, zooming, feedback on selection, select row/column from headers, config panel)

Papers

• [ ] retrieve open source pdf automatically
• [ ] upload paper?
• [ ] import web content with provenance?

Data Service

• [ ] integrate dojo - upload dataset
• [ ] sample data instead of whole thing for performance? (please see the related issue https://github.com/DARPA-ASKEM/TERArium/issues/372)
• [ ] summary stats for each column
• [ ] annotation with DKG (transpose Jataware code from React -> Vue)
• [ ] database migration script automated

Stakeholders

• Jataware
• Ahmed M.
• Daniel C.
• David G
• Yohann P.
• Pascale P.

[brandomr]

@YohannParis @dgauldie this looks good. The work we'll be doing related to the Data Service this sprint involves trying to refine the storage format/schema and trying to determine how best to represent geospatial information.

Currently we store CSVs in this format but want to assess whether we can rely on NetCDF instead. The primary reason for this is that we want to maintain performance as we move towards larger datasets for space weather AND we know that the TA3 teams are going to be using NetCDF as well. The challenge is that many COVID datasets are not gridded; instead they've got categorical place names (e.g. "Travis County, Texas, United States"). Additionally, we would like to be able to seamlessly combine features across datasets.

@mattprintz and @Sorrento110 are working on this analysis. Generally speaking, we can export data from the Data Service's internal format to whichever format the HMI prefers.

[amostafa-uncharted]

With regards to searching for Tables and Figures, an existing implementation that utilizes COSMOS is prototyped here: https://xdd.wisc.edu/set_visualizer/sets/askem-covid-demo/?query=vaccine&type=Figure but the current implementation for fetching Figures and Tables as Extractions is done through the new xDD ASKEM API. This recently evolving API only supports retrieving extractions for a specific document DOI and very recently has introduced a query param that seems to support extraction search, i.e. "query_all" (hence it is experimental and probably not working properly).

In addition to the aforementioned note, the data explorer will need a large plumping effort to get this feature added. I would confirm the priority of this feature and perhaps ask for a rough design first to decide on things such as the content of the matrix view and facets, which won't be available if this feature is implemented! @pascaleproulx fyi