Docstrings with SQL-like words in them break highlighting

[birdonfire]

There's a problem with the syntax highlighting when viewing this file: https://github.com/18F/rdbms-subsetter/blob/master/subsetter.py

Something in the docstring at the top of the document is causing the problem. Perhaps something to do with the RST markup?

The syntax highlighting on the GitHub page suffers the same problem.

There are no actual syntax errors in the document, and the application code runs just fine.

SublimeText's highlighter renders it fine.

[birdonfire]

If I lower-case the word "INSERTS" on line 43, it totally fixes the highlighting.

[birdonfire]

It seems that the highlighter interprets the upper case INSERTS as the beginning of a SQL block that isn't then closed.

<div class="line cursor-line" data-screen-row="42">
    <span class="source python">
        <span class="string quoted double block sql python">
            rdbms-subsetter only performs the
            INSERTS; it
            <span class="string quoted single sql">
                <span class="punctuation definition string begin sql">'</span>
                s your responsibility to set
            </span>
        </span>
    </span>
    <span class="invisible-character">¶</span>
</div>

If I lower-case INSERTS, this block now looks like:

<div class="line cursor-line" data-screen-row="42">
    <span class="source python">
        <span class="string quoted double block sql python">
            rdbms-subsetter only performs the inserts; it's your responsibility to set
        </span>
    </span>
    <span class="invisible-character">¶</span>
</div>

[amlinger]

I'm experiencing the same issue, where we are extracting docstrings for usage in our API documentation, and therefore we are using markdown formatting inside the docstrings:

"""Delete Resource
`DELETE http://some-url.com/resource`
"""
def destroy(self):
    pass

Which messes up the rest of the file (and here as well, where I expected pass to be highlighted, and DELETE not to be).

To me it seems as the rule named string.quoted.double.block.sql.python is the cause of this behaviour, and I've been fiddling around with it for a while but haven't really understood the grammar definitions yet.

It seems as the rule accepts additional characters apart from only whitespace between the intial """ specified by begin, and the beginning of the SQL statement specified by the keywords + whitespace, but have not come any farther than that.

[hvdklauw]

Any progress on this?

I just ran into the same problem.

Seems the syntax file starts the SQL context and only closes it when it encounters three double quotes, which basically eats them and leaves nothing for the commend to be closed.

[morenoh149]

I'm getting this for DELETE as well

):
    '''
    Accounts
    ###**Actions**:
      - `GET /:id/card`
      - `POST /:id/card`
      - `PUT /:id/`
      - `DELETE /:id/`
    '''
    serializers = {

highlights everything from Accounts to the end of the file.

[krzysztof-sikorski]

I just stumbled upon this, it seems that

"""'SELECT'"""

is enough to trigger that bug.

[dborstelmann]

I figured out the cause, and am working on a solution but would love ideas. It's because when the rule is applied for the triple quote sql strings, it runs include: source.sql inside the matched pattern. That is why the html classes that show on the text after a single ' or " come from the language-sql grammar. So the actual root cause is the language-sql grammar seeing the beginning of a sql string and then never finishing itself. If anybody knows how to ignore the internal SQL strings inside of certain parts of triple quoted strings, or knows how to make it so it can't override that outer """, let me know.

UPDATE: A proposed solution: Write a package on top of the python grammar that makes python use a slightly changed SQL grammar, but only in python files (so we don't mess with SQL everywhere else). That changed grammar only has one changed thing: SQL strings don't just end on ', but also \n. If people are okay with no multiline strings in SQL in python files then this solution would work perfectly. Let me know if anyone wants this and I can write it.

[jdw1996]

@dborstelmann Your proposed solution sounds reasonable to me, and I would definitely find it helpful.

[nckswt]

@dborstelmann have you managed to write up your solution yet? It would definitely be useful!

[dborstelmann]

@nckswt @jdw1996 Sorry I've been super busy recently, it looks the above reference will fix it? Does that work for you guys?

[winstliu]

@dborstelmann True, the above PR should fix it, but some things to be aware of:

1. It's probably going to take a while to be reviewed,
2. I'm not even sure if that will get merged,
3. We're probably still going to switch over to MagicPython in at least the short term

[jdw1996]

@dborstelmann I didn't look too closely into that PR, but on @50Wliu's suggestion, I've now switched to MagicPython.

[nckswt]

Swapping over to MagicPython to try it out as well. Thanks!

[dborstelmann]

Lemme know if you guys still want this, but for now I will assume MagicPython is a good stopgap.

[alisianoi]

Thank you, the highlight issue happened for me too because of a capitalized WITH in a docstring. Switching to MagicPython did the trick for me (just install and it works).

[Hellzed]

MagicPython helped me too ! (In my case a SOFT_DELETE in a docstring)

[ezarowny]

The SQL keywords can actually appear anywhere within a word within a docstring and cause trouble. Example: `ALL_ITEMS_WITH_DATA` breaks highlighting for me. Of interest, it breaks highlighting in a different manner than just ALL_ITEMS_WITH_DATA does.

Wrapping the word in backticks causes all the text in the file following the backtick-enclosed word to appear commented. Including the word alone causes any following SQL keywords in the docstring to become highlighted.

[goi42]

Just to pile on, I experience the same issue: MagicPython fixed it, but this really shouldn't be an issue...

[7inyu]

Experiencing the same issue here. Someday...someday...

[emptyopen]

Bumping...

[lorenzznerol]

Easy fix perhaps, not sure whether this 2015 issue is still alive: paste it below the docstring as a commented block.