question about how subscriptions are handled when metadata is removed

[woollattd]

small question

I notice that there has been some activity with metadata in recent months around the GISCs... so What happens when a user has subscribed to some harvested metadata that is later removed by the originating provider? I don't seem to be able to remove metadata manually on OpenWIS when there are subscriptions attached. Won't this cause catalogue discrepancies between the GISCs?

[lmika-bom]

Hi @woollattd. Had a check in the code and it looks like these metadata records will not be deleted. Following the code (I haven't actually tried it on a real server), once it has harvested records from the remote GISC, it will try to delete metadata records one by one. The first stage of the metadata deletion is to check whether there are any users subscribed to the metadata. If so, the deletion raises an exception, which will be added as a harvesting error but will not abort the process of deleting the remaining metadata records.

[woollattd]

That's right. That's what happens. I manually add some metadata; subscribed to it; tried to delete it - couldn't; removed subscription; can then manually remove metadata - But not sure if this would work with a metadata synch though since it's only looks at what's changed since the last time, and the 'exception' probably isn't taken into account in a later synch?

It seems clear to me that this will lead to the metadata getting out of synch with the true number from an originating GISC - leading to the whole global GISC catalogue system being a mess. There needs to be some mechanism to inform the user that this metadata is no longer available to subscribe to and that their subscription will be removed - since there's no point keeping it - no data will be produced for them to receive. It would also be useful to inform them of the metadata that is replacing it it, if there is any, so they can subscribe to that one instead.

I suspect this might be something for WMO to add guidance on. I can't find any about this anywhere. Or was it an original requirement created for OpenWIS?

[solson-nws]

All

Let me ask a follow up question. Right now, if a new metadata record gets created for the same weather product, does it do so using a different name? If so, could we simplify this process into the use of static metadata record names(using a static file nomenclature for the names). That way, when metadata records are updated, the subscription will pick this up and there would be no need to remove old metadata records or cancel old subscriptions. Just a thought.

Steve

On Fri, Oct 14, 2016 at 3:31 AM, Dom Woollatt notifications@github.com wrote:

That's right. That's what happens. I manually add some metadata; subscribed to it; tried to delete it - couldn't; removed subscription; can then manually remove metadata - But not sure if this would work with a metadata synch though since it's only looks at what's changed since the last time, and the 'exception' probably isn't taken into account in a later synch?

It seems clear to me that this will lead to the metadata getting out of synch with the true number from an originating GISC - leading to the whole global GISC catalogue system being a mess. There needs to be some mechanism to inform the user that this metadata is no longer available to subscribe to and that their subscription will be removed - since there's no point keeping it - no data will be produced for them to receive. It would also be useful to inform them of the metadata that is replacing it it, if there is any, so they can subscribe to that one instead.

I suspect this might be something for WMO to add guidance on. I can't find any about this anywhere. Or was it an original requirement created for OpenWIS?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenWIS/openwis/issues/211#issuecomment-253729476, or mute the thread https://github.com/notifications/unsubscribe-auth/APAjUa-bbjo8cDiuei8j-T9zsU9f7P_yks5qzy_BgaJpZM4KUkj7 .

[woollattd]

Hi Steve Thanks for your thoughts :-)

I'm not quite sure when that would happen as you say the the product already has metadata, why would it need another one? or do you mean updated rather than new?

Updates to the content of current metadata should have no effect on subscriptions. The 'urn' is linked to the data, and vice versa.

Metadata should only have one product (dataset) linked. It gets messy otherwise. I certainly wouldn't recommend having more than one metadata for the same dataset, as users might not realise and set up multiple subscriptions to the same data.

There will be a need to remove metadata from time to time - ie product is no longer produced or has been replaced with a completely new one (with a new name). The problem here is that the old metadata will be kept in the catalogue when it has at least one subscription linked to it. This means that this metadata can still be found and subscribed to by other users, but they would never receive any data. And the more of these there are the more the catalogues between the GISCs will become out of synch.

At this stage, I wouldn't want OpenWIS to automatically send replacement products. The end user might need to set up their end for any changes in product name, size etc.. , so it's best that they are notified. Also, if users are unaware, we could be sending them new data and cause problems for them with this data building up on their system, or failing, as no ingestion was set up for it. And the user would be wondering where the old data is. One more thing... they might not want the new one. ;-)

I believe we need some thought on how we intergrate this into OpenWIS. Maybe bring this issue up at WMO level as well, as there's no point OpenWIS covering this when others' systems might not. Like I say in my second post - i've seen no guidance on this.

Cheers Dom

[psgillis]

Please keep in mind that the metadata is informational regarding the actual data product. As such it rarely changes for an existing product set. Changes that would cause a change in the metadata would be, for instance, schedule change or origination. Changes such as that would require a subsequent change to the XML field in the corresponding metadat.

-pat

On Fri, Oct 14, 2016 at 10:19 AM, Dom Woollatt notifications@github.com wrote:

Hi Steve Thanks for your thoughts :-)

I'm not quite sure when that would happen as you say the the product already has metadata, why would it need another one? or do you mean updated rather than new?

Updates to the content of current metadata should have no effect on subscriptions. The 'urn' is linked to the data, and vice versa.

Metadata should only have one product (dataset) linked. It gets messy otherwise. I certainly wouldn't recommend having more than one metadata for the same dataset, as users might not realise and set up multiple subscriptions to the same data.

There will be a need to remove metadata from time to time - ie product is no longer produced or has been replaced with a completely new one (with a new name). The problem here is that the old metadata will be kept in the catalogue when it has at least one subscription linked to it. This means that this metadata can still be found and subscribed to by other users, but they would never receive any data. And the more of these there are the more the catalogues between the GISCs will become out of synch.

At this stage, I wouldn't want OpenWIS to automatically send replacement products. The end user might need to set up their end for any changes in product name, size etc.. , so it's best that they are notified. Also, if users are unaware, we could be sending them new data and cause problems for them with this data building up on their system, or failing, as no ingestion was set up for it. And the user would be wondering where the old data is. One more thing... they might not want the new one. ;-)

I believe we need some thought on how we intergrate this into OpenWIS. Maybe bring this issue up at WMO level as well, as there's no point OpenWIS covering this when others' systems might not. Like I say in my second post

• i've seen no guidance on this.

Cheers Dom

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenWIS/openwis/issues/211#issuecomment-253813402, or mute the thread https://github.com/notifications/unsubscribe-auth/AGYCr8AHUacJk4R9rk9-nYNxk0TU0TT8ks5qz49qgaJpZM4KUkj7 .

Patrick S. Gillis Program Lead for ISCS and WIS Programs NOAA/National Weather Service (301) 427-9689 patrick.gillis@noaa.gov WWW.Weather.gov/iscs

[solson-nws]

Dom:

I agree that some discussions need to happen, and thanks for your explanation. That clears up my misunderstandings. I understand the problem much better now. We could use the bi-weekly meeting forum for this type of discussion, or we could plan a webex session with just this topic. I'm happy to be in on the discussions no matter when they are!

Thanks!

Steve

On Fri, Oct 14, 2016 at 10:19 AM, Dom Woollatt notifications@github.com wrote:

Hi Steve Thanks for your thoughts :-)

I'm not quite sure when that would happen as you say the the product already has metadata, why would it need another one? or do you mean updated rather than new?

Updates to the content of current metadata should have no effect on subscriptions. The 'urn' is linked to the data, and vice versa.

Metadata should only have one product (dataset) linked. It gets messy otherwise. I certainly wouldn't recommend having more than one metadata for the same dataset, as users might not realise and set up multiple subscriptions to the same data.

There will be a need to remove metadata from time to time - ie product is no longer produced or has been replaced with a completely new one (with a new name). The problem here is that the old metadata will be kept in the catalogue when it has at least one subscription linked to it. This means that this metadata can still be found and subscribed to by other users, but they would never receive any data. And the more of these there are the more the catalogues between the GISCs will become out of synch.

At this stage, I wouldn't want OpenWIS to automatically send replacement products. The end user might need to set up their end for any changes in product name, size etc.. , so it's best that they are notified. Also, if users are unaware, we could be sending them new data and cause problems for them with this data building up on their system, or failing, as no ingestion was set up for it. And the user would be wondering where the old data is. One more thing... they might not want the new one. ;-)

I believe we need some thought on how we intergrate this into OpenWIS. Maybe bring this issue up at WMO level as well, as there's no point OpenWIS covering this when others' systems might not. Like I say in my second post

• i've seen no guidance on this.

Cheers Dom

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenWIS/openwis/issues/211#issuecomment-253813402, or mute the thread https://github.com/notifications/unsubscribe-auth/APAjUa5NzGfGdQwxzVtvoVUKpWoAT6atks5qz49rgaJpZM4KUkj7 .

[lmika-bom]

Hi @woollattd and @solson-nws,

I suspect this might be something for WMO to add guidance on. I can't find any about this anywhere. Or was it an original requirement created for OpenWIS?

I bet it was a consequence on how OpenWIS links subscriptions to metadata records (you cannot get data from OpenWIS without a record - hence the use of draft metadata) along with a lack of any clear guidelines as to how to decommission metadata from the WMO. It would be interesting to see how other implementations handle this case: I cannot imagine we would be the only ones that need to cater for it. Of course it all depends on how they manage their request/subscriptions.

One way we could handle this without addressing this uncertainly at the WMO level would be to do what Dom suggested and send out an automatic email to each of the subscribed users saying that data for this metadata is no longer available, remove the metadata and the associated subscriptions. The only potential problem with that is the case when the metadata might be deleted but the data feeds are still available, or the metadata is deleted by accident. We would loose the subscription information and the users will need to recreate them.

Another potential way we could handle this would be to remove the metadata from the catalogue, and then immediately create a draft metadata record and reassign the subscriptions to it. That way, the record would be out of the "public" catalogue but should data for those records still be ingested by OpenWIS (which I bet would happen even though it isn't meant to), these users will still be able to get them.

Anyway, something for the backlog.

Thanks,

Leon

[lmika-bom]

No real reason to fix this for v3, but might be worth thinking about for v4 as this is re-implemented.

[rogers492]

This issue was moved to OpenWIS/openwis4#56

[woollattd]

reopened since OpenWIS v3.x is in maintenance mode for next 6 years so this still applies