search

LSSTDESC / ComputingInfrastructure

Gathering place for CI - Computing and Infrastructure - issues
3 stars 1 forks source link

"DESC Notes" template, guidelines, use cases #24

Closed drphilmarshall closed 3 years ago

drphilmarshall commented 8 years ago

In https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/284 we're talking about writing up our results as short "DESC Notes" so that Chris W can cite them in his DC1 requirements doc. This could be a pathfinder for the "research notes" that the proposed Publication Policy refers to. To make a start, how about we put together a sensible little template and workflow, perhaps with some "Guidelines for DESC Note Authors", and try and help the Twinkles TF use it when documenting their results so far? Want to work on this with me, @heather999 ? Anyone else?

heather999 commented 8 years ago

Hi @drphilmarshall, yes that sounds possible! I found the proposed publication policy and am reading over the DESC notes portion. I've heard some bleeps about using a "document manager", such as DOCdb This is the type of thing we won't come to a decision on immediately - but I'd like to take a few moments today to see if there are any templates or guidelines we can follow that would make it easier to migrate our DESC notes in the future (if necessary!) For now, where do you plan to store these documents? In GitHub displayed via gh-pages? somewhere else? In the back of my mind, I wonder about LSST The Docs to store such files - but that may be something for later, and in the meantime we get set up elsewhere.

tony-johnson commented 8 years ago

Let me raise an issue which I have mentioned in the past, but which I think is important, namely searchability. My concern in particular is that if we use many different systems for storing DESC meeting minutes, documentation, issues, publications, internal notes, chats, etc, then the possibility of ever implementing a uniform search mechanism becomes very remote. I would rather see us using fewer technologies, even if not quite as functional, but with the possibility of providing a unified search tool.

heather999 commented 8 years ago

That is the ultimate purpose of LSST the Docs, as I understand it - to provide that searchability and serve as one stop shopping for content. What I remain unclear on, is how ready it is to serve this purpose at the present time. I will inquire. We did just receive a Twinkles PR 283 where @jonathansick has helped us get set up with LSST The Docs and we could attempt to test it.

I see two steps here.. short term.. provide a basic template for these DESC notes to use now, with the understanding it can be tweaked and modified as we move along. I am assuming we'll be using Markdown or ReStructuredText as our format. In the near future, we need to determine where these DESC notes will be made easily accessible.

Getting back to the template for a DESC note, I think this may come in handy as a possible guide: DM's Tech Notes

jonathansick commented 8 years ago

First off, I think DESC Notes are a great idea. There's certainly a huge need for communicating results in a manner that's faster and more flexible than arXiv/journals, more organized than emails, and doesn't fit into the paradigm of software documentation either. I introduced DM Technotes for much the same reasons as you've identified. Of course, DESC is free to engineer its own doc infrastructure, but I thought I'd share the roadmap for LSST DM docs. DESC and other collaborations are free to hop onto this roadmap and use any of the hosting resources we provide. You can also PR features into the doc infrastructure to help suit your needs (all our stuff is MIT open source). At the same time, this is a bit like driving down a highway while you're building it; things are bound to a bit rough and unfinished in the beginning.

One strategy is that DESC might roll-out its own lightweight solution that could be incorporated into some of the things I'll mention below once those platforms are in production. That way you'd be able to start writing and communicating right away, while realizing the benefits of DM's infrastructure later on without attempting to replicate it up-front. In general, we advocate developing docs as static files on GitHub; if you do that you'll be generally compatible with any service that DM provides.

Here's what we're working on:

At the LSST Project and Community Workshop next month I'll be speaking briefly in a keynote about the communication and doc services that DM can provide to science collaborations and we can talk further about how DM can help you (and I can tweak my development plans to meet your needs were possible). Having a unified doc approach across DM and science collaborations could free up a lot of effort for you. At the same time, don't feel pressured to use our services or be locked into our development timeline.

heather999 commented 8 years ago

@jonathansick I was meaning to ask you if your talk at the LSST Project and Community Workshop would be one of the talks broadcast for remote participants?

jonathansick commented 8 years ago

I think I'll be giving a 15 minute talk as part of a Wednesday plenary on DM products for the community. I'm not sure what the plans will be for recording talks, but I'd be happy to find a way to make sure it's broadcast. I'd also be happy to talk in person and hack on doc projects with anyone at LSST2016.

drphilmarshall commented 8 years ago

Thanks Jonathan - a great deal of this functionality is very interesting to us indeed, I think! To follow up on Tony's concern, one feature the DESC Research Notes will need to have is that only a fraction of them will be able to be made public - for scientific reasons some (or perhaps many) will need to be kept private until the parent journal paper is published. Meanwhile, we will still need to be able to cite them all, as we communicate with each other. Is this a problem for the LTD system? Can you propose a solution if it is?

BTW I think our notes will need to be named "Research Notes" rather than "Technical Notes", just because of our notes' target audience.

sethdigel commented 8 years ago

I'm interested to understand better how LaTeX Research Notes could be handled. In the example above for LDM-151, I see that the landing page (if that is the term) is the index.html file in the repository for LDM-151. What generates that file?

jonathansick commented 8 years ago

For LDM-151, I simply made an HTML page by hand: https://github.com/lsst/LDM-151/blob/draft/deployment/index.html. The Travis CI script (https://github.com/lsst/LDM-151/blob/draft/.travis.yml#L14) copies the built PDFs into the deployment directory where the cloned index.html resides and ltd-mason-travis ships those files up to LSST the Docs. A next step might be to render the HTML with something like Jinja2 templates to include information about the build date/branch/commit information. I've also been thinking of including an on-page PDF viewer (e.g. https://github.com/mozilla/pdf.js). My current focus hasn't been our LaTeX/PDF documents so I haven't had a chance to refine the PDF document presentation just yet.

sethdigel commented 8 years ago

Thanks for the explanation, @jonathansick. Those possibilities sound very good.

jonathansick commented 8 years ago

@drphilmarshall 🤔 I'll have to think about the embargo and auth use case. Private docs aren't in our roadmap for either LTD or DocHub. Not to say it can't be done, though. You could have a private DESC LTD server and instead of S3+fastly, service documents from a basic web server with your own auth service.

Unfortunately we don't have a great project-wide authentication & authorization service yet so I haven't been able to build against it (SQuaRE often uses GitHub OAuth against organization membership, which could work well for DESC too).

Another possibility is to look at off-the-shelf services like Authorea and Figshare. I haven't done this, but perhaps you could get a DOI for an embargoed research note on Figshare so that you can still cite the note.

drphilmarshall commented 8 years ago

I thought a little more about this. I think it might be OK to produce a web-visible listing of DESC Notes that includes titles and authors of all Notes, no matter whether they are visible or not, and just not provide the links to the private Note files - but that'll be a question for our pub board. For now we can use GitHub (via zenodo) to get a doi for each release of a repo - we could tag the repo each time one of its notes is "accepted" by the working group hosting the project, and then use that release's doi and Note file URL(s).

I have some limited experience with PDF viewing on GitHub - here's a review we wrote in a GitHub repo, whose PDF is displayed in a nice viewer (that enables the hyperlinks, crucially):

http://drphilmarshall.github.io/Ideas-for-Citizen-Science-in-Astronomy/

The code to make this page is here:

https://github.com/ewanmellor/gh-publisher

drphilmarshall commented 8 years ago

Some discussion aggregation for posterity:

drphilmarshall commented 8 years ago

@tony-johnson and I started jotting down some use cases for LSST DESC Notes, to help us have a good discussion with @jonathansick this week in Tucson. Feel free to add your own here.

drphilmarshall commented 8 years ago

Now in Tucson, I ran into @jonathansick just now and told him we were distilling our thoughts into a list of LSST DESC Note use cases, and would love to sit down with him sometime this week to discuss them. @heather999 @sethdigel watch this space: I'll find a good time to meet with Jonathan and let you know so you can join by BlueJeans if you want.

heather999 commented 8 years ago

Thanks @drphilmarshall, it would be great to join via BJs. Keeping my eyes on this issue :)

jonathansick commented 8 years ago

Hey @heather999 would 8 am PDT Tuesday (tomorrow) work for you on blue jeans?

heather999 commented 8 years ago

@jonathansick, 8 AM PDT tomorrow would work out just fine - thanks for the advance notice! :)

drphilmarshall commented 8 years ago

Great! Jonathan, lets meet for breakfast at 7:45am and find a quiet spot to chat with Heather.

On Mon, Aug 15, 2016 at 5:55 PM, Heather Kelly notifications@github.com wrote:

@jonathansick https://github.com/jonathansick, 8 AM PDT tomorrow would work out just fine - thanks for the advance notice! :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DarkEnergyScienceCollaboration/ComputingInfrastructure/issues/24#issuecomment-239973236, or mute the thread https://github.com/notifications/unsubscribe-auth/AArY9-3fWL8S4hNtewu4sY_1Wv4SHFiHks5qgQqVgaJpZM4JRdut .

sethdigel commented 8 years ago

Thanks for the opportunity to join you remotely. I could make Tuesday 8 am Pacific but any other day this week would be better.

Seth

-------- Original message -------- From: Heather Kelly notifications@github.com Date: 8/15/16 4:06 PM (GMT-08:00) To: DarkEnergyScienceCollaboration/ComputingInfrastructure ComputingInfrastructure@noreply.github.com Cc: Seth Digel digel@stanford.edu, Mention mention@noreply.github.com Subject: Re: [DarkEnergyScienceCollaboration/ComputingInfrastructure] "DESC Notes" template, guidelines, use cases (#24)

Thanks @drphilmarshallhttps://github.com/drphilmarshall, it would be great to join via BJs. Keeping my eyes on this issue :)

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/ComputingInfrastructure/issues/24#issuecomment-239955507, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFwZe_QG8TbXpNKxICfaUptCFHO5BlNQks5qgPDegaJpZM4JRdut.

heather999 commented 8 years ago

umm.. hmm.. so do we reschedule or are we going for it? actually do we have a BJ number?

sethdigel commented 8 years ago

I'm wondering the same thing.

jonathansick commented 8 years ago

Hi @heather999 and @sethdigel the URL is https://bluejeans.com/513427879/

drphilmarshall commented 8 years ago

Thanks for meeting with us this morning, @jonathansick! I am sure we will have more questions as we go along with our experiments, but it was great to be able to sit down and chat for an hour.

Your plans to enable LSST-wide elastic search of all project and science collaboration documentation makes me think we should be working on our metadata from the beginning. Presumably if you switch to json instead of yaml format you will have a fair bit of migration you will need to do on the LSST technotes, and so will be trying to script that, right? In which case we should work on producing yaml metadata as well, so we can take part in the migration when it happens too. Is this a good plan, or should we hold off on the metadata altogether for now? (The Twinkles notes folders only have template rst/md/ipynb files, no yaml files as yet.) Perhaps we should try and adapt your cookiecutter project to support DESC notes? Yesterday you suggested forking it and not expecting to ever re-merge - we can do that.

heather999 commented 3 years ago

From the Ops Manager: We are storing DESC Notes as Confluence attachments (with links to the attachments stored in PubDB. This means that in principle, we can search Notes using the search feature of Confluence to search attachments in the LSSTDESC space.