search

INTERMAGNET / wg-www-gins-data-formats

Repository to track working group discussions for WWW/Gins/Data Formats
2 stars 1 forks source link

Proposed workflow for INTERMAGNET WWW/Gins/Data Format Working Group discussions. #2

Open jmfee-usgs opened 5 years ago

jmfee-usgs commented 5 years ago

Proposed workflow for INTERMAGNET WWW/Gins/Data Format Working Group discussions.

1) Create issue in github project

2) Discussion takes place by adding issue comments.

3) Document the result of the discussion

sputnik-a commented 4 years ago

One suggestion: In case of discussion documents, such as #3, we could already start with a discussion file (e.g. using Markdown) that can be altered withing GIT during the discussion.

SimonFlower commented 4 years ago

I see the process described here as facilitating the initial discussion of a topic (and thus either supplementing or replacing the INTERMAGNET Discussion Documents that we've used in the past), but not replacing the next steps, which is where discussions and decisions move into other methods of documentation:

CharlesBlais commented 4 years ago

@sputnik-a Yes, I absolutely agree. Version control and user tracking of discussions is great.

@SimonFlower As you say, it proved efficient this year for INTERMAGNET meeting minutes (I am just doing copy+paste) of action and discussiong and as both you and Achim point, for tehcnical notes.

One challenge is non-tech savvy individuals. Git is second nature for most of us but some don't even understand the basics (its not straightforward). That being said, if everyone takes the effort to learn the basics, it should be good.

As for the technical manual, it can be maintained into its own repository and linked on the website. For example, https://intermagnet.github.io/metadata/ is https://github.com/INTERMAGNET/metadata. Plenty of opportunities of creating a collaborative environment.

I'll pass that along as a suggestion in the minutes.

CharlesBlais commented 4 years ago

On that note, I also propose we start a format for technical notes in markdown for the next meeting. What do you all think?

SimonFlower commented 4 years ago

I can see advantages to doing this, but since we are proposing changing the mechanism for how INTERMAGNET makes and records decisions, I think this should go to a plenary discussion.

SimonFlower commented 4 years ago

I certainly like the idea of the Technical Manual becoming a repository so that we can share out the work of updates. Are there any security concerns about people outside INTERMAGNET being able to access the technical manual while it's in draft form in the repository?

CharlesBlais commented 4 years ago

We can protect any documents from pull requests so that any pull needs to be reviewed by John Doe.

sputnik-a commented 4 years ago

Are there any security concerns about people outside INTERMAGNET being able to access the technical manual while it's in draft form in the repository?

There exists also the possibility to make a repository private, so that only collaborators can view its content.

SimonFlower commented 4 years ago

I think this would be useful (making the repository private) for the technical manual project?

jmfee-usgs commented 4 years ago

As a non-member, I would like to reference the manual even in draft form. I believe the value of transparency is underrated. It represents the consensus of INTERMAGNET, even in draft form, and is more up to date than v4. v4 remains the standard for IMOs until the new version is released and adopted.

None of the contents are sensitive, just preliminary, and updates are restricted to appropriate committee members. The repository should include at lease license, disclaimer, contribution guidelines, and any other documents describing the contents and practices. If part of the process is to ignore public comments for now, that can be mentioned in a document.

SimonFlower commented 4 years ago

I can see arguments for both sides (private and open) but I think that again, the decision needs to be made in a wider forum than this, as it affects other groups in INTERMAGNET. In this case, particularly the Technical Manual subcommittee should contribute to the discussion.

Khomutov-SY commented 3 years ago

My question is about header of iaga2002 files. If the observatory sends data in the iaga2002 format to GIN and writes comments in the header, then these comments are eventually lost. Can save these comments? For example, we write in the header the information about the source of the gaps and how we filled them... When a user requests our data from INTERMAGNET, this information is no longer there.

SimonFlower commented 3 years ago

At the Edinburgh GIN we store data in a database, not in any of the formats that the data is sent to us. This has a number of advantages: it is easy for us to handle any incoming data format; we can easily cope with fragments of day files and duplication of data, when a new format is introduced (such as Intermagnet CDF) we can make the format available for all data very quickly. The disadvantage is that we don't store the metadata from any of the formats.

It has been on my mind for some time to start a GIN metadata database alongside the data database, to collect the sort of metadata that Sergey is talking about. I know that many observatories put effort into their metadata and it is a shame that the Edinburgh GIN loses this metadata. I will put some thought into this issue (it also affects the CDF format, although I don't think anyone is delivering data to the GIN in CDF).

I don't think the other GINs work in this way. I think that metadata is preserved when sending data through them?

CharlesBlais commented 3 years ago

Just making sure we keep this subject on topic, I'll create a separate issue for meta information. See #11

CharlesBlais commented 3 years ago

Going back to the earlier, what does everyone think of this idea being sent to plenary? (Mostly small edits from @jmfee-usgs initial suggestion).

  1. Create issue in Github project of the committee (ex: https://github.com/INTERMAGNET/wg-www-gins-data-formats/issues) or any committee member puts a ticket on behalf of someone from the public. However, preference given to first option for easier tracking and contribution by the user. If the issue/item is of a private matter, it is placed in a private repository where committee members are given access.

  2. Discussion takes place by adding issue comments. Anyone with a github account can comment on issues and anyone given access to the private repository can comment on private issues.

  3. Document the result of the discussion in a document in the project. If consensus has emerged, changes to content within this repository should be submitted as a pull request. This may be in the form of documentation of the decision (preferrably markdown for easier updates). Alternatively this may result in other changes to content in the repository. The pull request description should reference the discussion issue (e.g. Fixes #XYZ), and after that has been merged the issue can be closed.

  4. If changes are required in other projects, or discussion results in multiple new issues for further discussion, create issues referencing the discussion or reference additional documentation before closing this issue.

  5. If it is decided to abandon the issue, a comment should be made describing the reason before it is closed.

SimonFlower commented 3 years ago

I'm happy with that. I think using issues in GitHub has worked well for discussions during the past two meetings and I can see it becoming a standard way of working, though it's worth remembering that it's easy to take up this way of working if you are already using Git in other work that you do, less easy if you are new to Git.

This would become a parallel way of working to using Intermagnet Discussion Documents and Technical Notes - it would be useful to provide a simple link between the Intermagnet document archive (http://geomag.bgs.ac.uk/intermagnet/) where discussion documents and technical notes are kept and the GitHub repository used for WWW/GIN/Data Format discussions, so that people are at least aware that discussion documents and technical notes are kept in more than one place.

Is this just a proposal for this subcommittee or a recommendation for others as well?

CharlesBlais commented 3 years ago

I would say a proposal for all. For example, IMO has started their private repository. Besides, finding commonality in operating introduces simplicity and less learning curves.

vmaury-ipgp commented 3 years ago

I think that's a really good idea. It's really easier to follow a GitHub dedicated topic than having plenty of emails. I think we could propose this to others because I found really difficult to follow discussions on NextCloud (no notifications, easy to remove text).