check iso version required by geonorge

ferrighi commented 3 years ago

geonorge seems to use iso 19115 instead of iso 19115-2, thus missing the MI_AcquisitionInformation block. According to the answer we get from geonorge we should modify the iso-to-genorge.xslt file and reindex all records.

ferrighi commented 3 years ago

@TAlonglong I am checking now the modifications. I have found some elements missing here: https://gitlab.met.no/mmd/s-enda-mmd-xml/-/blob/master/remote-sensing/satellite/metopb-avhrr-20201201155244-20201201160030.xml

You don't provide access_constraint which should be "Open"
We are missing the organization name which is "unknown" but it should be Norwegian Meteorological Institute
We have not followed up on the spatial_representation which is also mandatory. It should be grid in your case as we said. But we need to add it also in mmd, the PR is still pending.
provide Metadata author in personnel (this is the mandatory contact point for metadata in inspire)
provide Investigator in personnel (this will be mapped owner, as geonorge requires this role)
update vocabulary definitions using GCMDSK and NORTHEMES for the prefix and use this reference (look at the table for controlled vocabulary): https://adc.met.no/node/96 to add the keywords_vocabulary, i.e. prefix:name:uri

A attach the xml that is creating the inspire file sent to geonorge. https://app.zenhub.com/files/227606604/b3c513d9-b402-46f6-9ed4-ade6c2d5ab42/download

ferrighi commented 3 years ago

spatial representation is now merged into mmd master.

TAlonglong commented 3 years ago

I think I got spatial_representation and the keywords ok.

But I dont understand how the metadata author and investigator should be in the netcdf file. I can't find metadata author in acdd. Nor investigator.

Ah, but it should end up like this. hm

  <mmd:personnel>
    <mmd:role>Investigator</mmd:role>
    <mmd:name>Norwegian Meteorological Insitute</mmd:name>
    <mmd:email>post@met.no</mmd:email>
    <mmd:organisation>Norwegian Meteorological Insitute</mmd:organisation>
  </mmd:personnel>
  <mmd:personnel>
    <mmd:role>Metadata author</mmd:role>
    <mmd:name>DIVISION FOR OBSERVATION QUALITY AND DATA PROCESSING</mmd:name>
    <mmd:email>post@met.no</mmd:email>
    <mmd:organisation>Norwegian Meteorological Insitute</mmd:organisation>
  </mmd:personnel>

ferrighi commented 3 years ago

My suggestion is this:

1 creator, i.e. principal investigator (owner for genorge). These are all acdd elements

creator_name: Norwegian Meteo... creator_type: institution - please add it for future use creator_email: em1 creator_institution: Norwegian Meteo... creator_url: met.no

2 contributors -> all other roles - only the firs two are acdd, but inspire requires email and institution so we need to add them.

contributor_name: cr2, cr3 contributor_role: Metadata author, Technical contact (Multiple roles should be presented in the same order and number as the names in contributor_names.) contributor_email: em2, em3 -> this is already in py-mmd-tools contributor_institution: ins2, ins3 -> this is already in py-mmd-tools

does this make sense? do you agree @mortenwh ? do you want to add also contributor_type since we are at it?

mortenwh commented 3 years ago

Looks good! And yes - contributor_type is probably good to have as well :)

TAlonglong commented 3 years ago

Thanks @ferrighi ! I was thinking the same way, but was not sure if the comma separator used with the contributor would work. Will make some tests.

TAlonglong commented 3 years ago

OK, now my test netcdf gives this in the personnel part in the xml file:

  <mmd:personnel>
    <mmd:role>Investigator</mmd:role>
    <mmd:name>Norwegian Meteorological Institute</mmd:name>
    <mmd:email>post@met.no</mmd:email>
    <mmd:organisation>Norwegian Meteorological Institute</mmd:organisation>
  </mmd:personnel>
  <mmd:personnel>
    <mmd:role>Metadata author</mmd:role>
    <mmd:name>DIVISION FOR OBSERVATION QUALITY AND DATA PROCESSING</mmd:name>
    <mmd:email>post@met.no</mmd:email>
    <mmd:organisation>Norwegian Meteorological Institute</mmd:organisation>
  </mmd:personnel>
  <mmd:personnel>
    <mmd:role>Technical contact</mmd:role>
    <mmd:name>Norwegian Meteorological Institute</mmd:name>
    <mmd:email>post@met.no</mmd:email>
    <mmd:organisation>Norwegian Meteorological Institute</mmd:organisation>
  </mmd:personnel>

master version of py_mmd_tools is handeling this perfectly!

TAlonglong commented 3 years ago

xml file here @ferrighi

https://drive.google.com/file/d/1V6KfnGDkgGSjFWSa8JmVtrHvr8A6UeR1/view?usp=sharing

Please let me know.

ferrighi commented 3 years ago

I think you still miss access_constraint "Open" and the spatial_representation "grid". Also the title/abstract should not have :

Direct Broadcast data processed in satellite swath to L1C Direktesendte satellittdata prosessert i satellittsveip til L1C. so just provide the title in title and there should be title_no (summary_no) in the mmd tools to extract that.

TAlonglong commented 3 years ago

grep grid metopb-avhrr-20201201155244-20201201160030.xml 
  <mmd:spatial_representation>grid</mmd:spatial_representation>

So that should be ok.

Will look into the rest

TAlonglong commented 3 years ago

Hm either I have done something wrong or there si some kind of missunderstanding.

In the xml file I have I see for title:

grep title metopb-avhrr-20201201155244-20201201160030.xml 
  <mmd:title xml:lang="en">en:Direct Broadcast data processed in satellite swath to L1C;nor:Direktesendte satellittdata prosessert i satellittsveip til L1C.</mmd:title>
    <mmd:title>en:Direct Broadcast data processed in satellite swath to L1C;nor:Direktesendte satellittdata prosessert i satellittsveip til L1C.</mmd:title>

Looks like py_mmd_tools make two entries in the xml file. The netcdf input is

        :title_lang = "en" ;
        :title = "en:Direct Broadcast data processed in satellite swath to L1C;nor:Direktesendte satellittdata prosessert i satellittsveip til L1C." ;

Is this some problem with py_mmd_tools @mortenwh ?

TAlonglong commented 3 years ago

I have added :access_constraint = "Open" in the netcdf file, but as far as I can see my test does not bring this to the xml file.

the yaml file has it defined like this:

access_constraint:
  maxOccurs: '1'
  minOccurs: '0'

the xml template file has this

  {% if data['access_constraint'] %}
  <access_constraint>{{ data['access_constraint'] }}</access_constraint>
  {% endif %}

So it looks ok, but alas, no luck

mortenwh commented 3 years ago

access_constraint needs an acdd og acdd_ext attribute name. Should we just use acdd_ext: access_constraint? This should be defined in the yaml file.

TAlonglong commented 3 years ago

ah, sorry. I did not understood that.

mortenwh commented 3 years ago

I need to check regarding the xml output. I'll create a new issue, and work on that.

But the netcdf file should have the following attributes for the title:

:title = "Direct Broadcast data processed in satellite swath to L1C."
:title_lang = "en" ;
:title_no = "Direktesendte satellittdata prosessert i satellittsveip til L1C."

@ferrighi - I thought this was how we wanted it:

<mmd:title xml:lang="en">Direct Broadcast data processed in satellite swath to L1C</mmd:title>
<mmd:title xml:lang="no">Direktesendte satellittdata prosessert i satellittsveip til L1C.</mmd:title>

Can you give an example of how it should be?

TAlonglong commented 3 years ago

adding to mmd_elements.yaml so it becomes:

access_constraint:
  maxOccurs: '1'
  minOccurs: '0'
  acdd_ext: access_constraint

Gives the expected output.

TAlonglong commented 3 years ago

I need to check regarding the xml output. I'll create a new issue, and work on that.

But the netcdf file should have the following attributes for the title:
:title = "Direct Broadcast data processed in satellite swath to L1C."
:title_lang = "en" ;
:title_no = "Direktesendte satellittdata prosessert i satellittsveip til L1C."
Ah, so my netcdf is completely wrong. Sigh, we must have change this back and forth some times now.

I will test ...

ferrighi commented 3 years ago

ok, we got a reply from Lars-Inger. I have made the corrections he asked for. I will push the changes to mmd-to-genorge.xslt. I would say that once we have the final mmd from Trygve, we can index one record and ask to harvest it. If it's all good we proceed to full indexing and harvest. What do you think?

mortenwh commented 3 years ago

Yes, that sounds good. We can probably use the existing kubernetes setup for that, since we're still waiting for the mmd-ingestor. What do you think @aheimsbakk and @vkbo?

On Wed, May 12, 2021 at 10:05 AM Lara Ferrighi @.***> wrote:

ok, we got a reply from Lars-Inger. I have made the corrections he asked for. I will push the changes to mmd-to-genorge.xslt. I would say that once we have the final mmd from Trygve, we can index one record and ask to harvest it. If it's all good we proceed to full indexing and harvest. What do you think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/metno/S-ENDA-documentation/issues/254#issuecomment-839557839, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2UBIY6UGQISXMFLZPZGDTNIZGPANCNFSM44AQHKCQ .

-- Morten Wergeland Hansen, PhD Meteorologisk Institutt / Norwegian Meteorological Institute T.: (+47) 915 47 844

TAlonglong commented 3 years ago

OK.

Updated access_contraint the various personnel title and abstract. and fixed the keywords'

But still I think the title and abstract is not ok. What do you say @ferrighi. In your example in the first post you have nor. I have no What is correct? I don't remember what should go where.

https://drive.google.com/file/d/1V6KfnGDkgGSjFWSa8JmVtrHvr8A6UeR1/view?usp=sharing

ferrighi commented 3 years ago

I think this looks perfect. We might have messed around a bit before. The idea is that you provide it like this in mmd, and I provide "nor" to genorge. Which is in place already. I will test the conversion. Then push a branch and @mortenwh you can test indexing. Then we write to Lars-Inge.

ferrighi commented 3 years ago

yes, I've tested it and I get all green flags from the validator. The only different we have is that we are not providing the WMS data_access. Just to make sure...is that intentional?

TAlonglong commented 3 years ago

Thanks @ferrighi !

WMS, I might have forgotten to add that flag. I will check, just a sec.

TAlonglong commented 3 years ago

Yes, I forgot. Updated link here @ferrighi :

https://drive.google.com/file/d/1V6KfnGDkgGSjFWSa8JmVtrHvr8A6UeR1/view?usp=sharing

ferrighi commented 3 years ago

a PR (https://github.com/metno/mmd/pull/158) on mmd is ready to include modifications. A test file should now be indexed in csw and be harvested successfully from geonorge. If so, we can index all files.

ferrighi commented 3 years ago

https://app.zenhub.com/files/227606604/4b45d863-11fd-45ed-a883-5ff16f2670a8/download

This is the iso file. Although it would be better to use the mmd-to-geonorge.xslt from the mmd repo to make sure all is working.

mortenwh commented 3 years ago

All code in master branches should be ready for testing now. We need to check that geonorge can harvest the new metadata from our staging server before this issue can be closed.

mortenwh commented 3 years ago

Lars-Inge Arnevik confirmed that they can harvest from our csw endpoint today!

metno / S-ENDA-documentation

check iso version required by geonorge #254