Cell breakpoints aren't always highlighted

[leonelgalan]

Description:

Breakpoints are only highlighted when Untitled file is created after trying to open a ipynb. Saving this file to a py removes all breakpoints highlights. For troubleshooting I positioned the cursor between mark and down on the first cell and triggered "Editor: Log Cursor Scope".

Steps to Reproduce:

1. Click on ipynb file in Tree View.

Notice how %% and %% markdown are highlighted. Also Python is the selected grammar.

2. Save file with .py extension: foo.py

Python is still the selected grammar, but the highlights between cells are lost.

3. On its own the Hydrogen breakpoints appears to work:

Versions:

Which OS and which version of Hydrogen and Atom are you running?

• macOS 10.13.5
• Hydrogen 2.9.0

Atom    : 1.36.1
Electron: 2.0.18
Chrome  : 61.0.3163.100
Node    : 8.9.3

Plugins:

Do you have any Hydrogen plugins installed and active?

No Plugins.

Logs:

No warnings found on Atom's Developer Tools.

[elpollodiablo84]

I think I've found the problem: lately Atom switched to tree-sitter, a new parser to manage syntax highlighting. It's enabled by default from version 1.32. Disabling tree-sitter in "Settings > Core" solves this issue. Sadly, I don't think you can inject an "old style" grammar into a new one, so I don't think it will be easy to definitively fix this problem.

[BenRussert]

@leonelgalan thanks for the detailed issue!

If you wanted to dig into this (@leonelgalan, @elpollodiablo84, or anyone else looking), I would start with our breakpoints.cson file. See atom grammar creation docs for help with how this should work.

I have not looked into this deeply, so that is just a guess to start. Please comment if these dont end up helpful for finding the issue.

---

3. On its own the Hydrogen breakpoints appears to work

Can you clarify what you mean by "On its own"?

[lgeiger]

Here are some more informations on why this is currently broken: https://github.com/atom/atom/issues/18196

[kylebarron]

I think you've diagnosed the problem well. I added the main Jupyter notebook code unless it's changed in the last few months.

When you open a Jupyter notebook that has metadata declaring it to be a Python file, Hydrogen forces Python syntax highlighting, which normally wouldn't appear until you save the file. This lets you run code through Hydrogen before saving the file. Apparently the Python syntax highlighting that's applied here by Hydrogen is always the TextMate grammar, even when the tree sitter global setting is enabled. Then when you save the file, Atom reapplies syntax highlighting and chooses the tree sitter one.

So for consistent highlighting, the import-notebook code needs to accurately choose the right syntax highlighting to apply.

Otherwise, not seeing breakpoints in a tree-sitter grammar is another issue.

[leonelgalan]

@BenRussert, I meant when choosing "Hydrogen breakpoints" manually.

@kylebarron I agree that this are separate issues. For me, the biggest is making it work with the new tree-sitter grammars. For the other issue, a workaround is saving the file.

I'm over my head, but I went through the docs: https://flight-manual.atom.io/hacking-atom/sections/creating-a-grammar/; created leonelgalan/tree-sitter-hydrogen-breakpoints; and attempted to contribute to this package following the instructions in the Contributing Guide.

It's still not working, not sure where it's failing, but I'll keep trying to make it work.

tree-sitter-hydrogen-breakpoints/grammar.js

I though the grammar could be very simple: the sequence of '%% followed by anything except end of line. I found the simplest grammars I could find(Aerijo/tree-sitter-todo and tree-sitter/tree-sitter-embedded-template) and build my own based on what I found there.

module.exports = grammar({
  name: 'hydrogen_breakpoints',

  rules: {
    breakpoint: $ => repeat(
      $.directive
    ),

    directive: $ => seq(
      '%%',
      $._name
    ),

    _name: $ => /[^$]*/,
  }
});

hydrogen/grammars/breakpoints.cson

scopeName: 'hydrogen.breakpoints'
name: 'Hydrogen breakpoints'
type: 'tree-sitter'
parser: 'tree-sitter-hydrogen-breakpoints'
scopes:
  'breakpoint': 'entity.name.function'

hydrogen/lib/main.js

const Hydrogen = {
  // ...

  activate() {
   // ...

    atom.grammars.addInjectionPoint('source.python', {
      type: 'comment.line',
      language (commentLine) {
        return 'hydrogen.breakpoints'
      },
      content (commentLine) {
        return commentLine;
      }
    });

   // ...

[kylebarron]

@leonelgalan I think it's unfixable pending https://github.com/atom/atom/issues/18196

[wadethestealth]

@kylebarron Yes, however I wouldn't call this a bug. There is support for TextMate Grammars. There is just a lack of support for TreeSitter Grammars atm. The only way to add current support for TreeSitter Grammars is to create our own grammar for each language (not ideal or worthwhile).