Think on how to ease pyvideo data correction

[Daniel-at-github]

In pyvideo ETL process we have:

• Extraction: A youtube scrapper
• Transform: Manual correction
• Load: Git + Pelican

I think that the transform part have to be eased to attract new contributors.

Ideas:

• Use a local editor, like Lektor do. Could be lektor be used as local editor?
• Convert pyvideo/data from json to yaml. Example:

  
  copyright_text: 'Standard YouTube License'
  # Use :  | to enable clear (Restructured) text for the description, you only have to indent it

description: | Speaker: Anthony Scopatz

Xonsh is general purpose shell that combines Python and the best features of Bash, zsh, and fish. Relying only the standard library and PLY, the xonsh language is a strict superset of Python that compiles to a Python AST. The shell provides exciting features such as a rich history, tab completion from bash and man pages, syntax highlighting, auto-suggestion, foreign-function aliases and more!

Slides can be found at: https://speakerdeck.com/pycon2016 and https://github.com/PyCon/2016-slides

duration: 1837 id: 5067 language: eng recorded: '2016-05-31'

related_urls format is ...

related_urls:

• 'https://github.com/PyCon/2016-slides'
• 'https://speakerdeck.com/pycon2016' slug: anthony-scopatz-xonsh-pycon-2016 speakers:
• 'Anthony Scopatz' tags: thumbnail_url: 'https://i.ytimg.com/vi/uaje5I22kgE/maxresdefault.jpg' title: xonsh videos:
• {type: youtube, url: 'https://www.youtube.com/watch?v=uaje5I22kgE'}

  
  Yaml is more readable and supports comments.
  * Don't ask for the description and mark it as pending when is missing, leaving to the users to fill it in (giving them the event schedule URL).
  * Any other option that lowers the entry barrier.

[willkg]

When I was running PyVideo, we had people submit things in JSON format and generally, the blocker was bad tooling and not the file format. We did toss around switching to YAML, but didn't end up doing that because it didn't seem like a big enough win and it created different problems. We did add a better editor to Steve, but no one used it as far as I know except @codersquid and I. However, that was when PyVideo was based on Richard and we were mostly using JSON as a serialization format to import data into the db, so there are differences between this iteration and that one.

Even so, I think I'd spend some time finding out what's blocking people you want to be contributing to pyvideo. Are they eager to hand-edit files and are blocked by the file format? Are they blocked by better tooling and could care less about the file format?

If anyone is at PyCon US 2017, it's definitely worth doing a survey and talking to people in person there.

My intuition and past experiences suggest tooling will be the bigger win, but it's probably a bigger ongoing project. This is one of the reasons we spent time on Steve as a separate project--so people could fork and experiment and try multiple directions at the same time.

Hope that's helpful!

[redapple]

Was there a discussion on this at PyCon US 2017 in the end? Any minutes, conclusions or decisions? I'd definitely like to have YAML to fix reST descriptions when they fail to render (when I convert from HTML to .md to reST -- is there an HTML-to-rest conversion lib? I wasn't able to use pandoc for this for some reason). When I have to fix the description, what I currently do is:

• use cat videos/sometalks.json | jq -r .description
• fix the reST payload
• use python's json.dumps(payload)
• and put that in the JSON file back

If only for the literal string blocks using |, I'd switch to YAML for editing, or at least accepting the 2 file formats.

[Daniel-at-github]

Reviewing pull requests is more work that it should be:

• Replicate the pull request
  • git remote add the remote branch
  • git fetch --all --prune
  • Pull the remote branch
  • Render the pages make html && make serve
• Open all the pages involved (usually one event)
• For every page
  • Review the html
  • Editing json
  • Edit the one line but multiline description reST field
  • Render the pages again
  • Repeat if needs more corrections
• python tools/reserialize.py --all .

Yaml eases the pain points of:

• Errors in json
• Converts the one line description in multiline

The way of the yaml conversion involves the tasks:

• [ ] Rename all .json to .yaml (yaml is a superset of json)
• [ ] Refactorize tools/reserialize.py using pyaml instead fo json and use it in the data repository.
• [ ] Refactorize plugins/json_reader.py to read the data.
• [ ] Refactorize the tests, the schema and the rest of the scripts using json:
  • [ ] plugins/drop_no_publish.py
  • [ ] plugins/event_info.py
  • [ ] plugins/writer.py
  • [ ] data/tools/edit_rest.py
  • [ ] data/tools/fill_id_field.py
  • [ ] data/tools/parse_for_detail.py
  • [ ] data/tools/pull_related_urls.py
  • [ ] data/tools/utils.py
  • [ ] data/tools/veyepar_alternate_pull.py
  • [ ] data/tools/veyepar_pull.py
  • [ ] data/tools/video_statistics.py
  • [ ] data/tools/youtube.py
• Am I missing anything?
  • [ ] update the search engine (thanks for the comment @zerok)

If we want to make a tool, instead of converting to yaml, IMHO it should have:

• pull data by pull request number
• easy navigation (events, videos), by key
• Renderize to html
• Edit in yaml or json
• Test:
  • json schema
  • reST fields
• Save in reserialized json

Any ideas for such a tool? what python tecnology? Any functionality in excess or missing?

[zerok]

If we are moving to YAML we also have to update the search engine 🙂