search

ioos / ioos-atn-data

Code, documentation and issue tracking for ATN NCEI archiving
https://ioos.github.io/ioos-atn-data/
1 stars 0 forks source link

Acoustic tag tracking data flow diagram #77

Open MathewBiddle opened 7 months ago

MathewBiddle commented 7 months ago

a place to jot down what a data flow might look like for Acoustic tracking data.

jdpye commented 7 months ago

like, passive acoustic, or acoustic tags? if the latter, can I help with this?

jdpye commented 7 months ago

see https://ocean-tracking-network.github.io/node-manager-training/02_OTN_System_and%20_structure_and_Outputs%20/index.html for a discussion of the way OTN flows acoustic data from raw and unverified to verified and shareable.

MathewBiddle commented 7 months ago

Hey @jdpye! Thanks for hopping in here. For this issue, I'm looking at acoustic tags (passive acoustic monitoring (PAM) data flow are starting to be drafted at https://github.com/ioos/mbon-docs/issues/41). I'm trying to wrap my head around ATN's connection with OTN. @mmckinzie probably has thoughts to build out here.

In looking at the first diagram on that page you referenced, what does the OBIS box mean? Is that an OBIS node that everything gets bundled into? Does OTN have something to pull from previous to DwC alignment?

edit: clarifying the language for passive acoustics in this context

jdpye commented 7 months ago

The centralizing schema in the OTN database setup is named obis, has been since time immemorial, originally it mapped fairly well to the DwC terms we needed to populate, but more and more we have evolved past being able to pull straight from the OBIS entities into a DwC archive. We have mappers to do a good Event Core and Occurrence Core version of a project's held data, but the central schema is not the one-stop shop for it anymore.

MathewBiddle commented 5 months ago

Okay, so here is where I'm at. I'm having trouble figuring out how US base researchers would be able to archive at NOAA/NCEI as well as getting information into our federal discovery systems, but I think that's a conversation for another day. 


%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#007396',
      'primaryTextColor': '#fff',
      'primaryBorderColor': '#003087',
      'lineColor': '#003087',
      'secondaryColor': '#007396',
      'tertiaryColor': '#CCD1D1'
    },
   'flowchart': { 'curve': 'basis' }
  }
}%%

flowchart TD

A["Acoustic Tag"] 

B[("OTN
Regional Acoustic 
Node
or
OTN proper")]

C{{"Darwin Core
Alignment"}}

D[(NCEI)]

E[("OTN IPT
OBIS-oceantrack")]

F[/"MBON 
Data Portal"\]

G([OBIS])

H([GBIF])

I[("IOOS Data Catalog
(data.ioos.us)")]

J[(NOAA OneStop)]

K[(data.gov)]

L[("Commerce
Data Hub")]

M[/"IOC-UNESCO Harmful Algae Information System"\]

N[/"Infographics"\]

O[/"ATN Data
Portal"\]

A --> B
B --> C
C --> E
E --> EP
B --OTN ERDDAP\nhttps://members.oceantrack.org/erddap/index.html\nor\nOTN node\ndatabase\nconnection\nor\nOTN GeoServer -->EP
G --> EP
H --> EP
E --> G
E --> H

D --> FC
I --> FC

subgraph EP [Example Products]
O
N
F
M
end

subgraph FC [Federal Catalogs]
J
K
L
end

click D "https://www.ncei.noaa.gov" "NCEI" _blank
click E "https://members.oceantrack.org/ipt/" "OTN IPT" _blank
click F "https://mbon.ioos.us" "MBON" _blank
click G "https://obis.org" "OBIS" _blank
click H "https://gbif.org" "GBIF" _blank
click I "https://data.ioos.us" "IOOS Catalog" _blank
click J "https://data.noaa.gov/onestop/" "NOAA OneStop" _blank
click K "https://data.gov" "data.gov" _blank
click O "https://portal.atn.ioos.us/" "ATN" _blank
MathewBiddle commented 5 months ago

After a chat with @jdpye some modifications to the flow in how it could look in some future...


%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#007396',
      'primaryTextColor': '#fff',
      'primaryBorderColor': '#003087',
      'lineColor': '#003087',
      'secondaryColor': '#007396',
      'tertiaryColor': '#CCD1D1'
    },
   'flowchart': { 'curve': 'basis' }
  }
}%%

flowchart TD

A["Acoustic Tag"] 

B[("OTN
Regional Acoustic 
Node
or
OTN proper")]

C{{"Darwin Core
Alignment"}}

D[(NCEI)]

E[("OTN IPT
OBIS-oceantrack")]

P[("OTN ERDDAP\nor\nOTN node\ndatabase\nconnection\nor\nOTN GeoServer")]

F[/"MBON 
Data Portal"\]

G([OBIS])

H([GBIF])

I[("IOOS Data Catalog
(data.ioos.us)")]

J[(NOAA OneStop)]

K[(data.gov)]

L[("Commerce
Data Hub")]

M[/"IOC-UNESCO Harmful Algae Information System"\]

N[/"Infographics"\]

O[/"ATN Data
Portal"\]

A --> B
B --> C
B .-> D
C --> E
E --> G
E --> H

D --> FC
I --> FC

FC .-> EP

E .-> EP
B -->P
P .-> EP
G .-> EP
H .-> EP

subgraph EP [Example Products]
O
N
F
M
end

subgraph FC [U.S. Federal Catalogs]
J
K
L
end

click D "https://www.ncei.noaa.gov" "NCEI" _blank
click E "https://members.oceantrack.org/ipt/" "OTN IPT" _blank
click P "https://members.oceantrack.org/erddap/index.html" _blank
click F "https://mbon.ioos.us" "MBON" _blank
click G "https://obis.org" "OBIS" _blank
click H "https://gbif.org" "GBIF" _blank
click I "https://data.ioos.us" "IOOS Catalog" _blank
click J "https://data.noaa.gov/onestop/" "NOAA OneStop" _blank
click K "https://data.gov" "data.gov" _blank
click O "https://portal.atn.ioos.us/" "ATN" _blank
MathewBiddle commented 4 months ago

@jdpye @mmckinzie could you please review the above diagram? Are there specific example products/applications we could highlight here for acoustic tracking? eg. remora?

A note about linestyle, the dotted lines are hypothetical pathways. Solid lines are pathways that are currently functioning.

jdpye commented 4 months ago

I like having this diagram a lot! I would say there's been a lot of discussion and clamor for standing up an ATN Acoustic Data Mirror / FedNode between the regional OTN nodes and NCEI. And if we do that, it could in turn be the thing that feeds the ATN Data Portal, instead of the current state of affairs for ATN acoustics, which is acoustic project metadata coming from the OTN GeoServer and the OTN members portal site (via a scraper)

jdpye commented 4 months ago

I've got to revamp the ERDDAP situation a bit on the OTN side, but the GeoServer and the node-to-node connectivity are where things are happening in production right now.

MathewBiddle commented 4 months ago

standing up an ATN Acoustic Data Mirror / FedNode between the regional OTN nodes and NCEI.

Like this?


%%{
  init: {
    'theme': 'base',
    'themeVariables': {
      'primaryColor': '#007396',
      'primaryTextColor': '#fff',
      'primaryBorderColor': '#003087',
      'lineColor': '#003087',
      'secondaryColor': '#007396',
      'tertiaryColor': '#CCD1D1'
    },
   'flowchart': { 'curve': 'basis' }
  }
}%%

flowchart TD

A["Acoustic Tag"] 

B[("OTN
Regional Acoustic 
Node
or
OTN proper")]

Q[(ATN Acoustic Data Mirror\n/\nFedNode)]

C{{"Darwin Core
Alignment"}}

D[(NCEI)]

E[("OTN IPT
OBIS-oceantrack")]

P[("OTN ERDDAP\nor\nOTN node\ndatabase\nconnection\nor\nOTN GeoServer")]

F[/"MBON 
Data Portal"\]

G([OBIS])

H([GBIF])

I[("IOOS Data Catalog
(data.ioos.us)")]

J[(NOAA OneStop)]

K[(data.gov)]

L[("Commerce
Data Hub")]

M[/"IOC-UNESCO Harmful Algae Information System"\]

N[/"Infographics"\]

O[/"ATN Data
Portal"\]

A --> B
B --> C
B .-> Q
Q .-> D
B .-> D
C --> E
E --> G
E --> H

D --> FC
I --> FC

FC .-> EP

E .-> EP
B -->P
P .-> EP
G .-> EP
H .-> EP

subgraph EP [Example Products]
O
N
F
M
end

subgraph FC [U.S. Federal Catalogs]
J
K
L
end

click D "https://www.ncei.noaa.gov" "NCEI" _blank
click E "https://members.oceantrack.org/ipt/" "OTN IPT" _blank
click P "https://members.oceantrack.org/erddap/index.html" _blank
click F "https://mbon.ioos.us" "MBON" _blank
click G "https://obis.org" "OBIS" _blank
click H "https://gbif.org" "GBIF" _blank
click I "https://data.ioos.us" "IOOS Catalog" _blank
click J "https://data.noaa.gov/onestop/" "NOAA OneStop" _blank
click K "https://data.gov" "data.gov" _blank
click O "https://portal.atn.ioos.us/" "ATN" _blank

I'm wondering if we need to split out OTN proper vs OTN regional node at this point?

mmckinzie commented 4 months ago

@MathewBiddle You could separate OTN proper from OTN regional node, but data will likely flow into ATN node from both sources depending on where researchers originally submitted data/metadata. So, for simplicity also fine to leave grouped together, in my opinion. Only other thoughts are that when we stand up a federated node, ATN DAC will likely preform the DwC conversion and serve data to OBIS and then serve decimated data from OBIS to NCEI?? for U.S. based researchers. I'd also add a dashed line connecting ATN node to the products box as we'll likely harvest data/metadata from ATN node to create project pages and/or other data product that can be served out through the Marine Life data network portals.

MathewBiddle commented 2 months ago

How might we get acoustic tracking into the IOOS Data Catalog (https://data.ioos.us/)? I think we're sorting this out with satellite telemetry via ERDDAP which can be indexed by the IOOS catalog.

One requirement for IOOS is data are represented in the IOOS Data Catalog...