chore: include the grammar files in the npm package

[mikavilpas]

Include the grammar files in the npm package

I started to work on a parser for a markdown derived language used by presenterm. I am new to tree-sitter parser development, but I think what I want to do is reuse the grammar from this parser and essentially add a couple of things on top.

Currently, the grammar is not included in the npm package, so I can't extend it. I think this is because the files field in the package.json file does not include the grammar files.

The current contents of the package can be seen here:

https://www.npmjs.com/package/@tree-sitter-grammars/tree-sitter-markdown/v/0.2.1?activeTab=code

The fix

To fix this, I added the grammar files to the files field in the package.json file.

npm publish before the changes:

$ npm publish --dry-run
npm WARN publish npm auto-corrected some errors in your package.json when publishing.  Please run "npm pkg fix" to address these errors.
npm WARN publish errors corrected:
npm WARN publish "repository" was changed from a string to an object
npm WARN publish "repository.url" was normalized to "git+https://github.com/tree-sitter-grammars/tree-sitter-markdown.git"
npm notice
npm notice 📦  @tree-sitter-grammars/tree-sitter-markdown@0.2.1
npm notice === Tarball Contents ===
npm notice 1.1kB LICENSE
npm notice 4.6kB README.md
npm notice 582B  binding.gyp
npm notice 1.0kB bindings/node/binding.cc
npm notice 479B  bindings/node/index.d.ts
npm notice 328B  bindings/node/index.js
npm notice 38B   bindings/node/inline.js
npm notice 1.8kB package.json
npm notice === Tarball Details ===
npm notice name:          @tree-sitter-grammars/tree-sitter-markdown
npm notice version:       0.2.1
npm notice filename:      tree-sitter-grammars-tree-sitter-markdown-0.2.1.tgz
npm notice package size:  4.1 kB
npm notice unpacked size: 9.9 kB
npm notice shasum:        8a721c25ecafe065004b64bb57a2b0402afee910
npm notice integrity:     sha512-FeJnVcDba5dpQ[...]EjLn/FFF3edQw==
npm notice total files:   8
npm notice
npm WARN This command requires you to be logged in to https://registry.npmjs.org/ (dry-run)
npm notice Publishing to https://registry.npmjs.org/ with tag latest and public access (dry-run)
+ @tree-sitter-grammars/tree-sitter-markdown@0.2.1

npm publish after the changes:

npm notice
npm notice 📦  @tree-sitter-grammars/tree-sitter-markdown@0.2.1
npm notice === Tarball Contents ===
npm notice 1.1kB   LICENSE
npm notice 4.6kB   README.md
npm notice 582B    binding.gyp
npm notice 1.0kB   bindings/node/binding.cc
npm notice 479B    bindings/node/index.d.ts
npm notice 328B    bindings/node/index.js
npm notice 38B     bindings/node/inline.js
npm notice 5.6kB   common/grammar.js
npm notice 2.0kB   package.json
npm notice 22.6kB  tree-sitter-markdown-inline/grammar.js
npm notice 197.3kB tree-sitter-markdown-inline/src/grammar.json
npm notice 25.6kB  tree-sitter-markdown/grammar.js
npm notice 159.4kB tree-sitter-markdown/src/grammar.json
npm notice === Tarball Details ===
npm notice name:          @tree-sitter-grammars/tree-sitter-markdown
npm notice version:       0.2.1
npm notice filename:      tree-sitter-grammars-tree-sitter-markdown-0.2.1.tgz
npm notice package size:  42.2 kB
npm notice unpacked size: 420.7 kB
npm notice shasum:        44164c19d37b0356c6082ce69709dac793990865
npm notice integrity:     sha512-QyVX78sTNjdas[...]X/Wt0ime1PXgw==
npm notice total files:   13
npm notice
npm WARN This command requires you to be logged in to https://registry.npmjs.org/ (dry-run)
npm notice Publishing to https://registry.npmjs.org/ with tag latest and public access (dry-run)
+ @tree-sitter-grammars/tree-sitter-markdown@0.2.1

[clason]

You can't extend the grammar.json; you need to extend grammar.js (using node) directly. Look at how the cpp parser derives from c, or how the various shader languages derive from cpp.

[mikavilpas]

Ok, thanks for the tip. Do you know what the purpose of grammar.json is, then?

I checked and it seems like the cpp parser project exports it, too.

https://www.npmjs.com/package/tree-sitter-cpp?activeTab=code (src/grammar.json)

[clason]

Yes, it's a compact, machine-readable, representation of the grammar that tooling can use. For example, you can regenerate or build the parser from it via tree-sitter generate grammar.json (which is important in your case since you don't need to access the base grammar, which generating from grammar.js needs to import).

[clason]

In general, we are in the process of unifying the tooling (documentation, workflows, bindings, manifests) of all the grammars in this org based on improved upstream support; we just haven't gotten to this one yet. So I would ask you to wait with any such improvements that are likely to be overridden anyway.

[ObserverOfTime]

Fixed in 5520eaa328d41f6817b86c1da7b4d87c705a64a8 & 62516e8c78380e3b51d5b55727995d2c511436d8