D1 mn onboarding (mnlite init, plus scraping and testing of metadata)

[iannesbitt]

Member node onboarding

scripts streamlining this process

step 1: data gathering

• on sonode.dataone.org and so.test.dataone.org
  • simple wrapper around opersist
  • input from either a simple json edited by the user or direct user input -> json template
  • need sudo for mnlite service restart

script outline:

1. load/gather member node data (mn id, orcid, description, submitter, technical contact, sitemap urls, etc.)
2. . use member node identifier to initialize new node in opersist ("urn:node:mnTestBONARES" -> instance/nodes/mnTestBonares/)
3. look up subjects by orcid using CNIdentity.getSubjectInfo() and create if necessary (submitter, technical contact, node subject)
4. (if necessary) dump new json to instance/nodes/mnTestBONARES/node.json
5. system call to sudo systemctl restart mnlite
6. use scrapy with sitemap urls to crawl database
7. use pyshacl to test records against science-on-schema.org shape graph and raise/log error if non-compliant
8. (production) add update schedule to crontab pending outcome of step 7

[iannesbitt]

step 2: node registration and approval

• production and staging (cn-ucsb-1.dataone.org, cn-stage-ucsb-1.test.dataone.org)
  • will require a combination of *sh (for curl and dataone-approve-node calls) and python (for xml parsing)
  • need sudo for key management

script outline:

1. ...

[mbjones]

@iannesbitt I took a quick look at this, but didn't have tie to dive in. Can I suggect that you present the approach at a Thursday dev meeting to get feedback from me, @datadavev , @taojing2002 and others on the onboarding process as encapsulated here?

[iannesbitt]

@mbjones yes that sounds like a good idea. I will put it on the schedule for next week.

[iannesbitt]

@mbjones actually I will be out next Thursday so it'll have to wait until March 2.