Add headers to nomer dump

seltmann commented 2 years ago

Add the headers to the name file dumps in nomer.

jhpoelen commented 2 years ago

@seltmann thanks for the suggestion for adding the header.

Can you please provide preferred columns headers illustrated in an example?

seltmann commented 2 years ago

I dont know the meaning of all the column contents so I am unable to assign preferred column headers correctly. Here is a pass at trying to do so but I do not know that the meaning of the column headers actually matches the data.

Here is a suggestion to get us started, but we should discuss when we meet Friday

jhpoelen commented 2 years ago

to reflect the left/right (or provided/resolved names), suggest to prefix dwc: terms with provided/resolved.

For instance:

provided:dwc:taxonID	provided:dwc:ScientificName	relation:dwc:taxonomicStatus	resolved:dwc:taxonID	resolved:dwc:taxonName	resolved:dwc:taxonRank		resolved: resolved:dwc:HigherTaxon	resolved:HigherTaxonIDs		relation:dwc:nameAccordingToID
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum	Acamptopoeum argentinum	HAS_ACCEPTED_NAME	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum	Acamptopoeum argentinum	species		Animalia \| Arthropoda \| Insecta \| Hymenoptera \| Andrenidae \| Acamptopoeum argentinum	https://www.discoverlife.org/mp/20q?search=Animalia \| https://www.discoverlife.org/mp/20q?search=Arthropoda \| https://www.discoverlife.org/mp/20q?search=Insecta \| https://www.discoverlife.org/mp/20q?search=Hymenoptera \| https://www.discoverlife.org/mp/20q?search=Andrenidae \| https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum	kingdom \| phylum \| class \| order \| family \| species	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum

jhpoelen commented 2 years ago

so, a single line documents a citable name relation

(provided) -> (relation) -> (resolved)

where the properties of the provided/relation/resolved are captured in separate columns. The relation would include a citation like (name relationship according to ITIS, DiscoverLife . . . )

jhpoelen commented 2 years ago

In upcoming nomer version, you can include an header based on the input/output schemas using commands like:

$ nomer list discoverlife --include-header | head -n4

providedExternalId	providedName	relationName	resolvedExternalId	resolvedName	resolvedRank	resolvedPath	resolvedPathIds	resolvedPathNames	resolvedExternalUrl
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum	Acamptopoeum argentinum	HAS_ACCEPTED_NAME	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum	Acamptopoeum argentinum	species	Animalia \| Arthropoda \| Insecta \| Hymenoptera \| Andrenidae \| Acamptopoeum argentinum	https://www.discoverlife.org/mp/20q?search=Animalia \| https://www.discoverlife.org/mp/20q?search=Arthropoda \| https://www.discoverlife.org/mp/20q?search=Insecta \| https://www.discoverlife.org/mp/20q?search=Hymenoptera \| https://www.discoverlife.org/mp/20q?search=Andrenidae \| https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum	kingdom \| phylum \| class \| order \| family \| species	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui	Acamptopoeum calchaqui	HAS_ACCEPTED_NAME	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui	Acamptopoeum calchaqui	species	Animalia \| Arthropoda \| Insecta \| Hymenoptera \| Andrenidae \| Acamptopoeum calchaqui	https://www.discoverlife.org/mp/20q?search=Animalia \| https://www.discoverlife.org/mp/20q?search=Arthropoda \| https://www.discoverlife.org/mp/20q?search=Insecta \| https://www.discoverlife.org/mp/20q?search=Hymenoptera \| https://www.discoverlife.org/mp/20q?search=Andrenidae \| https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui	kingdom \| phylum \| class \| order \| family \| species	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense	Acamptopoeum colombiense	HAS_ACCEPTED_NAME	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense	Acamptopoeum colombiense	species	Animalia \| Arthropoda \| Insecta \| Hymenoptera \| Andrenidae \| Acamptopoeum colombiense	https://www.discoverlife.org/mp/20q?search=Animalia \| https://www.discoverlife.org/mp/20q?search=Arthropoda \| https://www.discoverlife.org/mp/20q?search=Insecta \| https://www.discoverlife.org/mp/20q?search=Hymenoptera \| https://www.discoverlife.org/mp/20q?search=Andrenidae \| https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense	kingdom \| phylum \| class \| order \| family \| species	https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense

Note that I have not yet introduced DwC terms associated with the column names due to ambiguity of mappings across existing mappers. Nomer supports non-taxonomic mappers also.

I imagine that translating the specific dumps into some DwC-like taxonomic scheme would be possible with the provided descriptive column names.

jhpoelen commented 2 years ago

@seltmann first pass at header functionality available in just released Nomer v0.2.9 . Please review.

seltmann commented 2 years ago

@jhpoelen

updated to nomer version 0.2.9
nomer list discoverlife --include-header | head -n4

Header names look accurate based on above discussion. Some tabs are missing between formatted data between providedName and relationName For example:

providedExternalId | providedName | relationName -- | -- | -- https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum | Acamptopoeum argentinumHAS_ACCEPTED_NAME | https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui | Acamptopoeum calchaqui | HAS_ACCEPTED_NAME https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense | Acamptopoeum colombiensHAS_ACCEPTED_NAME | https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense

seltmann commented 2 years ago

nomer list discoverlife --include-header | head -n4 includes resolvedCommonNames, resolvedPath, resolvedPathIds, resolvedPathNames, resolvedExternalUrl, resolvedThumbnailUrl

nomer list --properties my.properties discoverlife does not include resolvedCommonNames, resolvedPath, resolvedPathIds, resolvedPathNames, resolvedExternalUrl, resolvedThumbnailUrl

curious if this is on purpose?

jhpoelen commented 2 years ago

@seltmann thanks for taking the time to review.

For some reason, I was unable to reproduce your results.

Here's what I did:

# install 0.2.9 
$ nomer version
0.2.9
# clean cache, just in case some old cached taxon files remained
$ nomer clean
...
$ nomer list discoverlife --include-header | head -n4 > withHeader.tsv
...

With attached withHeader.tsv (I added a txt extension for github to accept it)

withHeader.tsv.txt

opening the file in LibreOffice calc, gets me the expected results (see attached screenshot).

Screenshot from 2021-11-24 14-38-19

How did you capture the output of the nomer command?

jhpoelen commented 2 years ago

Also re:

nomer list --properties my.properties discoverlife

can you please share your my.properties file?

seltmann commented 2 years ago

nomer version 0.2.9

clean cache I did not clean cache last time, so did this time

nomer list discoverlife --include-header | head -n4 > withHeader.tsv

I imported into Google Sheets from .tsv file and this looks fine.

seltmann commented 2 years ago

cat my.properties nomer.append.schema.output.example.taxon.rank.order=[{"column":0,"type":"path.order.id"},{"column": 1,"type":"path.order.name"},{"column": 2,"type":"path.order"}] nomer.append.schema.output=[{"column":0,"type":"externalId"},{"column": 1,"type":"name"},{"column": 2,"type":"authorship"},{"column":3,"type":"rank"}] nomer.schema.input=[{"column":0,"type":"externalId"},{"column": 1,"type":"name"},{"column": 2,"type":"authorship"},{"column": 3, "type":"rank"}]

jhpoelen commented 2 years ago

@seltmann thanks!

with your my.properties, I was able to do:

$ $ nomer list --properties my.properties --include-header discoverlife | head -n4
[main] INFO org.globalbioticinteractions.nomer.match.TermMatcherRegistry - using matcher [discoverlife-taxon]
providedExternalId  providedName    providedAuthorship  providedRank    relationName    resolvedExternalId  resolvedName    resolvedAuthorship  resolvedRank
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum  Acamptopoeum argentinum (Friese, 1906)  species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum  Acamptopoeum argentinum(Friese, 1906)   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui   Acamptopoeum calchaqui  Compagnucci, 2004   species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui   Acamptopoeum calchaqui  Compagnucci, 2004   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense Acamptopoeum colombiense    Shinn, 1965 species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense Acamptopoeum colombiense    Shinn, 1965 species

and

$ nomer list --properties my.properties --include-header discoverlife | head -n4 > withCustomHeader.tsv

with attached result (with appended .txt for github)

withCustomHeader.tsv.txt

Weird. Can you reproduce?

jhpoelen commented 2 years ago

@seltmann weird as in: I cannot reproduce your missing header, and I am noticing the expected custom headers show up.

jhpoelen commented 2 years ago

btw - I noticed how my tab characters disappeared on copy-pasting from the terminal, but not in the redirected file output. Perhaps this explains the missing tabs from before.

seltmann commented 2 years ago

All is fine regarding the tabs now.

Still seeing difference between the headers, but I think this is by design?

nomer list discoverlife --include-header | head -n4 includes resolvedCommonNames, resolvedPath, resolvedPathIds, resolvedPathNames, resolvedExternalUrl, resolvedThumbnailUrl

nomer list --properties my.properties discoverlife does not include resolvedCommonNames, resolvedPath, resolvedPathIds, resolvedPathNames, resolvedExternalUrl, resolvedThumbnailUrl

jhpoelen commented 2 years ago

After running:

$ nomer list --properties my.properties --include-header discoverlife | head -n4
$ head withCustomHeader.tsv 
providedExternalId  providedName    providedAuthorship  providedRank    relationName    resolvedExternalId  resolvedName    resolvedAuthorship  resolvedRank
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum  Acamptopoeum argentinum (Friese, 1906)  species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+argentinum  Acamptopoeum argentinum(Friese, 1906)   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui   Acamptopoeum calchaqui  Compagnucci, 2004   species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+calchaqui   Acamptopoeum calchaqui  Compagnucci, 2004   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense Acamptopoeum colombiense    Shinn, 1965 species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense Acamptopoeum colombiense    Shinn, 1965 species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiensis    Acamptopoeum colombiensis   Shinn, 1965 species SYNONYM_OF  https://www.discoverlife.org/mp/20q?search=Acamptopoeum+colombiense Acamptopoeum colombienseShinn, 1965 species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+fernandezi  Acamptopoeum fernandezi Gonzalez, 2004  species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+fernandezi  Acamptopoeum fernandeziGonzalez, 2004   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+inauratum   Acamptopoeum inauratum  (Cockerell, 1926)   species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+inauratum   Acamptopoeum inauratum  (Cockerell, 1926)   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+melanogaster    Acamptopoeum melanogaster   Compagnucci, 2004   species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+melanogaster    Acamptopoeum melanogaster   Compagnucci, 2004   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+nigritarse  Acamptopoeum nigritarse (Vachal, 1909)  species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+nigritarse  Acamptopoeum nigritarse(Vachal, 1909)   species
https://www.discoverlife.org/mp/20q?search=Acamptopoeum+prinii  Acamptopoeum prinii (Holmberg, 1884)    species HAS_ACCEPTED_NAME   https://www.discoverlife.org/mp/20q?search=Acamptopoeum+prinii  Acamptopoeum prinii (Holmberg, 1884)    species

$ nomer list --properties my.properties --include-header discoverlife | head -n4 > withCustomHeader.tsv
...
$ nomer list --properties my.properties discoverlife | head -n4 > withoutCustomHeader.tsv
...

with attached withCustomHeader.tsv / withoutCustomHeader.tsv

withoutCustomHeader.tsv.txt withCustomHeader.tsv.txt

As far as I can tell, I don't see any unexpected results: the specified input and output schemas are used, and include things like provided authorship and resolved authorship.

Please confirm.

seltmann commented 2 years ago

@jhpoelen you are correct, those using --properties my.properties are the same and correct.

I was commenting on the output without using my.properties. my.properties adds columns (authors, ranks) but also removes columns resolvedCommonNames, resolvedPath, resolvedPathIds, resolvedPathNames, resolvedExternalUrl, resolvedThumbnailUrl

jhpoelen commented 2 years ago

@seltmann thanks for clarifying and for being patient with me.

Yes, if you specify a non-default schema, you'd have to explicitly include all the desired columns.

Am curious to hear thoughts on how to make this schema business a little more intuitive (if needed). Otherwise, please let me know if you have any remaining desires / comments on the current --include-header functionality introduced in v0.2.9 .

seltmann commented 2 years ago

@jhpoelen can I get a list of other properties I can add to my.properties? I see that nomer -p allows me to use a custom properties, but can I configure this to also include resolvedPath for example. Thanks!

jhpoelen commented 2 years ago

@seltmann adding a new issue for your feature request to list all columns that can be added to input/output schemas.

@seltmann are there any remaining issues regarding to this specific issue (add header to nomer dump/list) ?

jhpoelen commented 2 years ago

See issue #67 for feature request related to schema terms.

globalbioticinteractions / nomer

Add headers to nomer dump #54