URGENT Some relaton-data-* repos should keep data folder

[andrew2net]

Most of relaton-data- repos regenerate all files in data dir every time they run, but some repos update only the last changes or don't generate any files at all. For example, relaton-data-iec is scheduled to fetch only the last updates, we need to keep previously fetched in this case. A full update can be run manually and all the files in the data dir should be removed in this case. The relaton-data-iho doesn't generate any files in the data folder at all https://github.com/relaton/relaton-data-iho/issues/22 We need to check all other relaton-data- repos if they need to keep the data folder.

This issue causes issues: https://github.com/relaton/relaton-iec/issues/51 and https://github.com/relaton/relaton-iec/issues/52

[CAMOBAP]

crawler updated, I will double check repos

@andrew2net should I revert last commits for relaton-data-iec or relaton-data-iho ?

[CAMOBAP]

@andrew2net I need your help, to confirm that data directoy should not be removed for:

• relaton-data-ids - Could you please also check https://github.com/relaton/relaton-data-ids/pull/3)
• relaton-data-itu-r
• relaton-data-calconnect
• relaton-data-3gpp
• relaton-data-ietf

Should not delete data (confirmed by @andrew2net previously):

• relaton-data-iec - should not remove data automatically
• relaton-data-iho - should not remove data automatically

The crawler.rb removes data in repos below (and it looks right):

• relaton-data-bipm
• relaton-data-nist
• relaton-data-w3c
• relaton-data-ecma
• relaton-data-oasis
• relaton-data-rfcs
• relaton-data-rfcsubseries
• relaton-data-ieee
• relaton-data-iana

Empty or manually updated repos:

• relaton-data-iso - manually updated
• relaton-model-jis - manually updated
• relaton-data-xsf - empty

Outdated crawler.rb (@CAMOBAP to prepare relevant PRs):

• relaton-data-ogc - should remove data in crawler.rb, I will prepare separate PR
• relaton-data-cie - remove data in crawler.rb (@CAMOBAP to update crawler.rb to avoid unnecessary git commits in it)

[andrew2net]

crawler updated, I will double check repos

@andrew2net should I revert last commits for relaton-data-iec or relaton-data-iho ?

@CAMOBAP I've regenerated data in the relaton-data-iec and disabled the schedule temporarily, don't touch it. The relaton-data-iho needs to be reverted.

[andrew2net]

We shouldn't remove the data directory for the following data repositories:

• relaton-data-itu-r I'm going to refactor the data-fetcher script later then the data dir should be removed.
• relaton-data-3gpp has its own logic to detect if the data should be cleared or not

We don't use the relaton-data-ietf anymore so its schedule should be disabled.

The relaton-data-iso and relaton-data-xsf are not in use yet. The relaton-model-jis is not a dataset. It's a JIS data model. We shouldn't do anything with the repo in this context.

I'm curious, why do we need to move the data-removing command from repos to the support?

[CAMOBAP]

I'm curious, why do we need to move the data-removing command from repos to the support?

I was my mistake because I assumed that all data repos should do data remove before update

It will not do remove anymore

[CAMOBAP]

All data repos doublechecked, and all repos that should keep data directory is keeped and and last commits reverted (for those repos where it was need)