Supporting vue (Was: require new features with {scope})

[Fmajor]

for scope, now we have these flags

{scope=push}
{scope=ref}
{scope=pop}
{scope=clear}
{scope=set}
{placeholder}

In my custom syntax file, i need to parse the same pattern in different way with different context (or different stack status), so i come up with some new features with {scope}, like

{scope=replace}
    replace the top of stack (pop and push, not clear and push as '{scope=set}')
{scope=push(n)}
    push current match nth times
{scope=pop(n)}
    pop current match nth times

also, i'd like to use some boolean operation for the scope, like

{scope-top-kind=blabla}, {scope-top-kind!=blabla}
{scope-level=blabla}, {scope-level>blabla}, {scope-level<blabla}
    the --regex-X will run only when the expression are true

i'm wondering if someone else have the same requirements.

[masatake]

Interesting idea but it needs consideration.

{scope=replace}
    replace the top of stack (pop and push, not clear and push as '{scope=set}')

Implementing this is not difficult. I wonder {scope=pop}{scope=push} does the same as {scope=replace}

{scope=push(n)}
    push current match nth times

Ctags can push a language object which is tagged to the scope stack. Pushing nth substring directly to the stack is not possible. Consider scope entry has not only name but also kind. kind is not specified in your notation.

BTW, matchTagPattern() in main/lregex.c is the function where scope stack is used. Let's think about the following code of imaginary language:

var Parent::item;

I guess you may want to make the following tag:

item  ipnut.X /^var Parent::item;/;" scope:???:Parent

We have to fill ???.

I know how useful the feature you propose is. However, tool fill the ???, I have to implement the way to handle reference tag in regex parser. See https://github.com/universal-ctags/ctags/pull/1260

With the reference tag, you can capture "Parent" in the code as a reference tag. A reference tag has a kind. So we can fill ??? with the kind.

I'm working on implementing the way to capture reference tags via regex parser very slowly. However, it will take time.

Your proosal may be implemented on that.

Do you have any target language? I would like to see.

. an example input file (shorter is better)

. expected tags output

. IDEAL command line options to make the output from the input.

You can use any imaginary options in the last item.

BTW, matchTagPattern() is the function in which the scope stack is uesd.

[Fmajor]

i am writing Vue, like this (i have delete all the function body..)

<script>
import _ from 'underscore'
import MTableRow from './m-table-row'
let eachColumn = {
  'data': [],  // do not want to match this!
  'subfields': {},
}

function abyb (a, b) {
   .....
}

export default {
  name: 'm-table',
  props: {
    'data': {
      'type': [Object],
      'default': function () {
        return {
          'columns': [],
          'fixColumnCount': null
        }
      }
    }
  },
  data () { // want to match this
    return {
      'addable': false,
      'deletable': false
    }
  },
  computed: {
    style () {
    },
    headerIDStr: {
      cache: false,
      get () {
      }
    }
  },
  'methods': {
    initData () {
    },
    getColumn (index) {
    },
    getColumnShow (index) {
    },
    updateColumnSort () {
    },
    getData (data, index) {
    },
    updateRowSort (sortBy) {
    },
    onMouseOverHeader () {
    }
  },
  created () {
    this.initData()
  },
  components: {MTableRow}
}
</script>

the name, data, props, computed, methods, created in this context ( ==> export default {} ) are things i want to display, like this

--regex-vue=/^ {2}((name|'name')[^,]*)/\1/p,property/{scope=ref}
--regex-vue=/^ {2}((data) \(|('data'):|(data):)/\2\3\4/p,property/{scope=replace}
--regex-vue=/^ {2}((props) \(|('props'):|(props):)/\2\3\4/p,property/{scope=replace}
--regex-vue=/^ {2}((computed) \(|('computed'):|(computed):)/\2\3\4/p,property/{scope=replace}
--regex-vue=/^ {2}((methods) \(|('methods'):|(methods):)/\2\3\4/p,property/{scope=replace}
--regex-vue=/^ {2}((created) \(|('created'):|(created):)/\2\3\4/p,property/{scope=replace}
--regex-vue=/^ {2}((components) \(|('components'):|(components):)/\2\3\4/p,property/{scope=replace}
--regex-vue=/^ {4}(([a-zA-Z0-9_]*) \(|('[a-zA-Z0-9_]*'):|([a-zA-Z0-9_]*):)/\2\3\4/p,property/{scope=ref}

i just want to match the "data ()" in the export default {}, not before that in "let eachColumn = {}", so i want the feature of boolean expression to make a constrain, like

--regex-vue=/^(export default)/\1/e,export/{scope=push}
--regex-vue=/^ {2}((name|'name')[^,]*)/\1/p,property/{scope=ref}{scope-top-kind=export}

[Fmajor]

also, i tried {scope=pop}{scope=push}, it act only as {scope=push}

[b4n]

Looks like a job for the JavaScript parser for it to emit something for the export statements. Would that work, or is it too much a problem because the JS is embedded in something else CTags doesn't (yet?) forward that part to the JS parser?

[masatake]

Would that work, or is it too much a problem because the JS is embedded in something else CTags doesn't (yet?) forward that part to the JS parser?

It is possible to run js parser areas which are embedded in another language like html.

As far as seeing https://vuejs.org/images/vue-component.png, One needs to write a vue parser which detects areas and applies JS parser(or css parser) to the areas.

Writing the vue parser in C language is one of approach. More interesting one is extending regex mline regex parser to allow wriging following command line.

--_mline-regex-Vue=/<script>(.|[\n])</script>///{area-start=1start}{area-end=1end}{guest-parser=JavsScript}

More studying vue is needed.

[masatake]

After thinking again, we don't have to extend the mline-regex parser right now. Even writing in C language, picking the areas (<script>...</script>) is not difficult. I did the same in yacc parser.

[Fmajor]

actually i do not want to run a JavaScript parser in the area, that will list too many symbols, i want to focus on the overview structure of the .vue file, and list all keys in the export default {} dict which have 3 format

export default {
  'name': 'test', // format1, this is a item with value
   props: { // format2, this is a item with value, but without quote
      'subkey': 'test'
   },
   data () { // format3, this is a function
       return {
           'result': 'test'
       }
   }
}

if we have format1 and format2(a object key)
    if the value is not a dict, extract it directly
    elif the value is a dict, extract it recursively
if we have format3 (a function)
    extract its return value (if it's a dict, extract recursively)

i don't know if it's possible without write a custom parser and only with improved regex-Vue command

=================update================ i think i can do the above things in the following logic (with several new command i want)

# first detect export default environment (use scope=replace because we are already in the <script> environment)
--regex-vue=/^export default/export default/e,export/{scope=replace}
# i want to parse only in the export environment with {if-stack-include-kind=e}
# if matched, we want to go into the { } environment either for a dict or a function
# so we need a way to correctly pop the stack after leaving the environment
# the method is push stack now, and do not push the next '{', pop all '}', and we will have the right stack status
# {skip} means not record this match
--regex-vue=/^ *(([a-zA-Z0-9_]*) \(|('[a-zA-Z0-9_]*'):|([a-zA-Z0-9_]*):)/\2\3\4/p,property/{if-stack-include-kind=e}{scope=push}{let-ignoreNext=1}{else}{skip}
--regex-vue=/\{//i,ignore/{if-stack-include-kind=e}{if-ignoreNext-ne=1}{scope=push}{else}{let-ignoreNext=0}{else}{skip}
--regex-vue=/\}//i,ignore/{if-stack-include-kind=e}{scope=pop}{else}{skip}

that is to add variables and if expression into the --regex syntax, i thing it will help a lot and make the ctags much more powerful

[Fmajor]

i use vim+tagbar a lot, but i just got to use ctags in this custom way yesterday. i find it a lot help to navigate vim in well structured file(e.g: automatically detect some keywords), so i want more features (like boolean operation during reg match, or even add variables in the --regex variable to help resolve complex context)

here is what i get now, it's not totally right, but still help me a lot to show me the over structure and navigate around it.

[Fmajor]

If the requirement are too complex, i'd like to try to write a custom parser, is there any demo or doc for that?

[codebrainz]

If it were me, I'd just hack the Vue code to make it emit ctags format, since it already knows how to parse the single file components and embedded languages.

[Fmajor]

hi @codebrainz

1. how to parse the single file components and embedded languages? just as what @masatake said?

   Writing the vue parser in C language is one of approach. More interesting one is extending regex mline regex parser to allow wriging following command line. --_mline-regex-Vue=/<script>(.|[\n])</script>///{area-start=1start}{area-end=1end}{guest-parser=JavsScript}

2. what does it mean to 'hack the Vue code to make it emit ctags format'?

[codebrainz]

how to parse the single file components and embedded languages? just as what @masatake said?

Probably, I'm the wrong person to ask, I only ever wrote a very simple parser for a simple language for the old ctags.

what does it mean to 'hack the Vue code to make it emit ctags format'?

I mean to modify the front-end code of Vue.js and make it only emit ctags' simple tags format to a file rather than doing all the other stuff Vue.js does after the front-end. I have no idea how practical that would be, but on the surface it seems simpler.

[Fmajor]

Yes, you are right. i use Webpack to handle my project, so maybe i can write a preload plugin to generate ctag file using javascript, it's much more easier, thanks.

[Fmajor]

i read the ctags documents, it says

A regex-based parser is inherently line-oriented (i.e. the entire tag must be recognizable from looking at a single line) and context-insensitive (i.e the generation of the tag is entirely based upon when the regular expression matches a single line).

the universal-ctags add {scope} flag so that we can record the context during parsing, i think the things i talk above are to use the context variable and make the regex-based parser context-sensitive, that will make the parser much more powerful

as far as i understand, the parser try all the regex one by one for each line, so i think we can add some 'if' flag or 'set variable' flag in the syntax, the command should run before or after doing actual regex match

1. 'if' flag decide whether or not to record the match and whether or not to change the stack
2. 'set' flag give variable new value, which may be used in later 'if'

like this

--regex-X=/pattern1/match/n,name/{if-y-eq=1}{set-x=1}{scope=push}{record}{else}{record}
--regex-X=/pattern2/match/n,name/{if-x-gt=2}{set-x=0}{scope=pop}{record}{else}{skip}
convert to:

for eachline in file:
    for eachPattern in allPatterns:
        match = parse(eachPattern, eachline)
        if match:
            ...

# for pattern1, the code in ... may like
    if y==1:
        x = 1
        stack.push(match)
        record(match)
    else:
        record(match)

# for pattern2, the code in ... may like
    if x>2:
       x = 0
       stack.pop()
       record(match)
    else:
       continue

that is to make the {flags} a small language with only 'if' expression and basic variable assignment and operation

[masatake]

Of course, I would like to add the code for parsing javascript source code using vue framework to ctags.

I think adding vue parsr as a sub parser of javascript parser is the best. Reducing the information can be done after parsing. Give me time. I'm catching up the discussion and learning vue.

[Fmajor]

@masatake thanks a lot!

Anyway, i still thinks it will make ctags much more powerful adding the "if" and "set" feature(i'd like to use it on some formatted file to generate navigation flags), since it will make the parser context-sensitive. This feature may cost some time to develop, you can think about it later.

Also as you said, adding the {scope=replace} flag is not difficult, so please consider to develop this feature first when you have time.

[codebrainz]

I think adding vue parsr as a sub parser of javascript parser is the best.

I have no idea if this makes sense in the context of ctags parsers, but a single-file view component is almost identical to a regular HTML document containing embedded CSS and JS. Does it make sense to improve the HTML parser to somehow be able to handle CSS and JSS (using subparser)?

[masatake]

@codebrainz, you may correct. Using HTML as a host parser will be better. Anyway, I think we can do very interesting thing with existing code.

[masatake]

https://vue-loader.vuejs.org/en/start/spec.html

@codebrainz, do you think we should run html parser for .vue file?

My idea is introducing a small vue parser. vue parser may run html parser for <template>...</template> areas, JavaScript parser for <script>...</script> areas, and css parser for <style>...</style> areas.

[masatake]

@Fmajor, my understanding is that what you want is the top level keys in export default block. JavaScript parser has ability to capture keys. So many code for implementing what you want is in ctags. What we have to do is specializing JavaScript parser to Vue.

export default. Is this special phrase of Vue or standard phrase of JavaScript?

[codebrainz]

@masatake it's part of JavaScript proper, and I don't believe it's the only way to achieve the same thing for Vue.

[masatake]

Oh, @b4n already wrote about "export". If JavaScript parser handling the block well, all parts of ctags may work fine.

export is something like prototype in C. I like using reference tag with "unknown" kind and ... "exported" role.

Anyway a special kind for it is needed. Maybe it is what @Fmajor wants.

When thinking about scope, things are not simple. If a::b is exported in c, I wonder which scope b should have: a or c. Here b is taged as a reference tag. In natural langauage, I can say b defined b is exported from c. Or c::a::b?

2017/10/20 午後2:15 "Matthew Brush" notifications@github.com:

@masatake https://github.com/masatake it's part of JavaScript proper https://developer.mozilla.org/en-US/docs/web/javascript/reference/statements/export, and I don't believe it's the only way to achieve the same thing for Vue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/universal-ctags/ctags/issues/1577#issuecomment-338108136, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEtFTv5GsERS-SSyxhGV4IrXBRscLyAks5suCyAgaJpZM4P3arD .

[masatake]

Surprisingly a function can be defiend in export block... So using unknown/exported role/kind combination doesn't make sense (or it not enough.).

[masatake]

@b4n, ctags vue parser I implemented can push an area inside <script>...</script> to JavaScript parser. I will make a branch for vue.

So could you consider how JavaScript parser deals with the export default block. @Fmajor has interest only the sub set of the output of JavaScript parser. Such filtering can be done in later.

1. VueParser calls JavaScript parser as a guest parser.
2. JavaScript parser parsers export default block well.
3. JavaScript calls VueJavaScript parser as sub parser.

VueJavaScript subparser picks up only the name, data, props, computed, methods, created keys.

[Fmajor]

I tried to apply the latest version ctags install from homebrew on some js file, but it failed to extract the "export default" part

let me explain my requests (and why i need it) in more detail. one month ago, i began to learn web front-end developing and start to write javascript code. Before that, i only write python and c code. I use vim and i have my costum folding plugin, so i can always quickly nagivate to the function/variables i want, i actually do not use ctags at that time.

But things are different when i change to .vue (javascript).

a .vue file are usually like

<template>
  <!-- some html code -->
  <div> ... </div>
  <!-- some html code with v-command (like v-if, v-for) and injected js command -->
  <div v-if="showTag === true">
    <div v-for="eachItem in data"> ... </div>
  </div>
</template>

<script>
// local functions and variables
var a = 1, b = 2
function test () {}
var d = function () {}
var e = () => {}

export default {
  name: 'name of a component',
   props: {
     'data': {
       'type': [Object],
       'default': function () {
         return {
           'columns': [],
           'fixColumnCount': null
         }
       }
     }
   },
  data () {
    return {
      'show': false
    }
  },
  computed: {
    style () {
    },
  },
  'methods': { // this is usually a LARGE dict and will take 50% of the total file
    someMethod0 () {
    },
    someMethod1 () {
    },
    someMethod0 () {
    }
  },
  created () {
  },
  components: {MTableRow}
}
</script>

<style>
</style>

vue file descript properties of a vue component, we care only about the overall structure of it

my final purpose is to quickly nagivate to the variable/functions i'm working on, in a vue file, those things are

1. some vue command in the part like v-if, v-for (because we have injected javascript code here)
2. style definitions in the