`file-types` section in `package.json` incomplete

[sogaiu]

Currently, clojure-mode.el has the following section:

  (add-to-list 'auto-mode-alist
               '("\\.\\(clj\\|cljd\\|dtm\\|edn\\)\\'" . clojure-mode))
  (add-to-list 'auto-mode-alist '("\\.cljc\\'" . clojurec-mode))
  (add-to-list 'auto-mode-alist '("\\.cljs\\'" . clojurescript-mode))
  ;; boot build scripts are Clojure source files
  (add-to-list 'auto-mode-alist '("\\(?:build\\|profile\\)\\.boot\\'" . clojure-mode))
  ;; babashka scripts are Clojure source files
  (add-to-list 'interpreter-mode-alist '("bb" . clojure-mode))
  ;; nbb scripts are ClojureScript source files
  (add-to-list 'interpreter-mode-alist '("nbb" . clojurescript-mode)))

via: https://github.com/clojure-emacs/clojure-mode/blob/3453cd229b412227aaffd1dc2870fa8fa213c5b1/clojure-mode.el#L3221-L3230

The current package.json has the following section: https://github.com/sogaiu/tree-sitter-clojure/blob/262d6d60f39f0f77b3dd08da8ec895bd5a044416/package.json#L21-L26

May be there are some things worth adding.

AFAIU, one thing this information affects is the tree-sitter cli's scanning (as described briefly here). I am not very clear on how this works, but I wonder if there could be issues if other non-Clojure grammars use any of the same file extensions.

[sogaiu]

I think cljd may refer to ClojureDart.

I'm not sure yet, but I think they may have some syntactic constructs that are not valid in Clojure or ClojureScript:

/(List String) is the ClojureDart pendant of Dart List and is in fact a tagged literal producing ^{:type-params [String]} List. Thus parametrized types are symbols as usual.

via; https://github.com/Tensegritics/ClojureDart/blob/main/doc/differences.md#parametrized-types

There might be other things that are different too (see the page linked to) -- the one above is just something I became aware of looking through their core.cljd file.

Update: According to cgrand:

As for ClojureDart, I don't believe you need to change anything to the grammar: ClojureDart has made no change to the reader (to not break cljc). We are just using / for tagged literals but your grammar handles it just fine.

So my statement eariler:

I think they may have some syntactic constructs that are not valid in Clojure or ClojureScript

is not correct. However, as can be seen in #46, our grammar wasn't handling everything appropriately.

[sogaiu]

On a side note, I know that there is at least one type of thing in ClojureCLR that won't work with tree-sitter-clojure. It's documented here:

Fully-qualified type names can contain an assembly identifier, which involves spaces and commas. Thus, fully-qualified type names cannot be represented as symbols. Generic type names include angle brackets and backquotes, and thus also are not representable as symbols. In fact, CLR typenames can contain arbitrary characters. Backslashes can escape characters that do have special meaning in the typename syntax (comma, plus, ampersand, asterisk, left and right square bracket, left and right angle bracket, backslash).

ClojureCLR extends the reader syntax for symbols to accommodate CLR type name by the mechanism of vertical bar quoting.

Some examples from the aforementioned page:

(namespace 'ab|cd/ef|gh)               ;=> nil
(name 'ab|cd/ef|gh)                    ;=> "abcd/efgh"
(namespace 'ab/cd|ef/gh|ij)            ;=> "ab"
(name 'ab/cd|ef/gh|ij)                 ;=> "cdef/ghij"

Note that unlike ClojureDart and ClojureScript, AFAIU, ClojureCLR does NOT use a different file extension to distinguish itself from Clojure. They both use clj.

Update: As clarified here, it turns out ClojureCLR can also use cljr.

[sogaiu]

Here's another list of extensions courtesy of helix editor:

file-types = ["clj", "cljs", "cljc", "clje", "cljr", "cljx", "edn", "boot"]

I think cljx might be referring to this: https://github.com/lynaghk/cljx -- I thought that was obsolete (and the README claims "deprecated"). I've never tested against such files and I'm not familiar with them.

I hadn't seen clje before -- perhaps it's Clojure for Erlang (clojerl). Seems possible: https://github.com/clojerl/clojerl/tree/master/src/clj/clojure. No idea if there is any divergence from JVM Clojure syntactically.

~Haven't found anything on cljr yet.~ See this comment regarding cljr.

[dannyfreeman]

I always thought cljr was for clojure CLR :)

Does the modified reader for clojure CLR cause the grammar to break? That seems like a feature that would be rarely used (does typical C# code produce assemblies with all those weird characters??). If the parser still works it might be worth not touching at the risk of making the parser even more complicated.

[sogaiu]

I always thought cljr was for clojure CLR :)

Apparently you were not alone: https://github.com/helix-editor/helix/pull/3387#issue-1335072560

As a former user speaking, I don't believe that was the case, but perhaps things have changed. May be we'll get a response to this.

[sogaiu]

That seems like a feature that would be rarely used (does typical C# code produce assemblies with all those weird characters??).

I don't have any data on how frequently things are used -- I'm pretty sure this extension to the syntax wasn't added lightly (and I think RH may be aware of this change).

Perhaps I should have chosen a different sample from that page.

Here's one that's much more likely to appear:

|System.Collections.Generic.IList`1[System.Int32]|

Here's some text from the same page regarding the above example:

This is the official CLR way of referring to the type that would be referred to in C# (with using) by IList.

Regarding:

Does the modified reader for clojure CLR cause the grammar to break?

I guess I can try but I doubt it will work.

Ok, here's what I get for the example:

(source [0, 0] - [1, 0]
  (sym_lit [0, 0] - [0, 33]
    name: (sym_name [0, 0] - [0, 33]))
  (syn_quoting_lit [0, 33] - [0, 35]
    value: (num_lit [0, 34] - [0, 35]))
  (vec_lit [0, 35] - [0, 49]
    value: (sym_lit [0, 36] - [0, 48]
      name: (sym_name [0, 36] - [0, 48])))
  (sym_lit [0, 49] - [0, 50]
    name: (sym_name [0, 49] - [0, 50])))

Seems to not work.

---

AFAIU, there aren't a large number of ClojureCLR users out there (if there are, they are hiding well) and the last I checked, the current maintainer is in the midst of rewriting from scratch using F# (not sure whether he's still doing this).

It doesn't seem like an urgent matter and it might be better done in a separate grammar.

Perhaps it is possible to handle this sort of thing using an external scanner though.

[dannyfreeman]

An extension grammar might work well for detecting these symbols, but it may be difficult for editors to use when the language seems to be using the same file extensions, how should it know when I file is meant to be executed on the CLR or the JVM?

FWIW, that specific symbol |System.Collections.Generic.IList`1[System.Int32]| causes clj-kondo errors and Emacs clojure-mode has issues with it as well. If no one has complained about that then I doubt it will become an issue with this grammar

[sogaiu]

An extension grammar might work well for detecting these symbols, but it may be difficult for editors to use when the language seems to be using the same file extensions, how should it know when I file is meant to be executed on the CLR or the JVM?

I didn't have difficulties in practice as I tended to manually manage my REPL connections, but there is a trick you can do with the REPL and reader conditionals to determine what sort of Clojure is sitting at the other end.

I agree it would be far less confusing if a different file extension were to be used. May be there is still time for that to change.

(On a side note, it looks like some support for ClojureCLR was added to inf-clojure last summer: https://github.com/clojure-emacs/inf-clojure/issues/202)

FWIW, that specific symbol |System.Collections.Generic.IList`1[System.Int32]| causes clj-kondo errors and Emacs clojure-mode has issues with it as well.

Thanks for checking.

If no one has complained about that then I doubt it will become an issue with this grammar.

Perhaps not any time soon anyway :)

[IGJoshua]

For clarity, .cljr is for the ClojureCLR port. That is a less-documented feature, but it is a part of the core runtime of ClojureCLR that it will look for .cljr files first when finding clojure files to load so that cross-platform libraries can be distributed without issue.

https://github.com/clojure/clojure-clr/blob/master/Clojure/Clojure/Lib/RT.cs#L3229

[sogaiu]

@IGJoshua Appreciate you taking the time to respond and explaining!

[sogaiu]

Re: ClojureDart - cgrand mentioned here that:

I don't believe you need to change anything to the grammar: ClojureDart has made no change to the reader (to not break cljc). We are just using / for tagged literals but your grammar handles it just fine.

So perhaps we can add cljd.

I suppose finding some collection of sample code and doing some testing first might not be a bad idea.

---

Below is an example of it working:

$ echo "#/(List String)" | tree-sitter parse -
(source [0, 0] - [1, 0]
  (tagged_or_ctor_lit [0, 0] - [0, 15]
    tag: (sym_lit [0, 1] - [0, 2]
      name: (sym_name [0, 1] - [0, 2]))
    value: (list_lit [0, 2] - [0, 15]
      value: (sym_lit [0, 3] - [0, 7]
        name: (sym_name [0, 3] - [0, 7]))
      value: (sym_lit [0, 8] - [0, 14]
        name: (sym_name [0, 8] - [0, 14])))))

This is using ahlinc's alpha that has parsing via stdin built in.

[NoahTheDuke]

This is using ahlinc's alpha that has parsing via stdin built

That is super cool! Nice find.

[dannyfreeman]

I think adding the cljd (and cljr probably) extension is a good move. Is there anything stopping us from doing so? I'm not too worried about the impact it would have on other things that might use those extensions. It just seems like a very remote edge case that will never come up.

[sogaiu]

Thanks for your comments.

I'm thinking to try to find some .cljd files to do some tests first [1]. We have very little actual usage experience ATM.

I'm not so sure about .cljr files -- possibly it's ok if we explicitly mention that we don't support ClojureCLR's pipe symbol construct, but again, I haven't tested much so if we were going to do this, I'd prefer that we test first.

As far as I'm aware, technically, this whole file extension thing only affects the tree-sitter cli (but may be that's not true) so I'm also not too concerned about anything else that might happen to use the extensions.

There is what people might read into it though, so I think making some appropriate statements might be preferrablle to be less "misleading".

---

[1] This search shows some results here at least.

[sogaiu]

Here's an initial list:

https://github.com/amjil/flutter-counter https://github.com/amjil/horizontal_scroll_picker https://github.com/amjil/mongol-anabapa-list https://github.com/amjil/mongol-date-picker https://github.com/amjil/horizontal-pull-to-refresh https://github.com/amjil/mongol-timeline https://github.com/amjil/mongol-virtual-keyboard https://github.com/ampersanda/clojuredart-cli https://github.com/ampersanda/clojuredart-counterapp https://github.com/CFiggers/flutter-animate-transition https://github.com/CFiggers/flutter-clojuredart-hello-world https://github.com/CFiggers/flutter-layout-tutorial https://github.com/CFiggers/flutter-navigate-demo https://github.com/CFiggers/flutter-persist-sqlite https://github.com/CFiggers/flutter-tabs-demo https://github.com/CFiggers/flutter-themes-demo https://github.com/cjbarre/dart-game-lab https://github.com/cjbarre/dart-lab https://github.com/clojurestream/clojuredart-workshop https://github.com/D00mch/ClojureDartTeaExample https://github.com/dupuchba/ryanapp https://github.com/ericdallo/clojuredart-sample https://github.com/kennytilton/flutter-counter https://github.com/kennytilton/flutter-mx https://github.com/lambdina/ifs https://github.com/lambdina/weather_neumorphism_ui https://github.com/PEZ/clojuredart-quickstart-root-problem https://github.com/prestancedesign/clojuredart-recipe-app https://github.com/Tensegritics/ClojureDart https://github.com/vinmaster/clojuredart-example

[sogaiu]

@dannyfreeman May be we can adapt the babashka script you wrote for clojars fetching to get these things (though I guess we might want to use git?) and make a babashka task for it plus parsing to test (a bit along these lines).

[sogaiu]

Looks like org.clojure/tools.gitlibs is usable from babashka (inserted lines to make output more readable):

$ bb
Babashka v1.1.174-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (require '[babashka.deps :as deps])
nil

user=> (deps/add-deps '{:deps {org.clojure/tools.gitlibs {:mvn/version "2.5.190"}}})
Clojure tools not yet in expected location: /home/user/.deps.clj/1.11.1.1208/ClojureTools/clojure-tools-1.11.1.1208.jar
Downloading https://download.clojure.org/install/clojure-tools-1.11.1.1208.zip to /home/user/.deps.clj/1.11.1.1208/ClojureTools/tools.zip
Unzipping /home/user/.deps.clj/1.11.1.1208/ClojureTools/tools.zip ...
Successfully installed clojure tools!
Downloading: org/clojure/tools.gitlibs/2.5.190/tools.gitlibs-2.5.190.pom from central
Downloading: org/clojure/tools.gitlibs/2.5.190/tools.gitlibs-2.5.190.jar from central
nil

user=> (require '[clojure.tools.gitlibs :as gl])
nil

user=> (def repo-url "https://github.com/clojure/spec.alpha.git")
#'user/repo-url

user=> (gl/resolve repo-url "739c1af5")
"739c1af56dae621aedf1bb282025a0d676eff713"

user=> (gl/procure repo-url 'org.clojure/spec.alpha "739c1af")
Checking out: https://github.com/clojure/spec.alpha.git at 739c1af
"/home/user/.gitlibs/libs/org.clojure/spec.alpha/739c1af56dae621aedf1bb282025a0d676eff713"

[sogaiu]

It doesn't look like it's worth the trouble to use org.clojure/tools.gitlibs for this. Quite awkward for just trying to clone a repository [1].

Instead, the following might be fine:

$ bb
Babashka v1.1.174-SNAPSHOT REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (require '[babashka.tasks :as t])
nil

user=> (t/shell "git clone https://github.com/CFiggers/flutter-animate-transition")
Cloning into 'flutter-animate-transition'...
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 0), reused 10 (delta 0), pack-reused 0
Receiving objects: 100% (10/10), done.
{:proc #object[java.lang.ProcessImpl 0x3632ce77 "Process[pid=64091, exitValue=0]"], :exit 0, :in #object[java.lang.ProcessBuilder$NullOutputStream 0x413fda5f "java.lang.ProcessBuilder$NullOutputStream@413fda5f"], :out #object[java.lang.ProcessBuilder$NullInputStream 0x18889aff "java.lang.ProcessBuilder$NullInputStream@18889aff"], :err #object[java.lang.ProcessBuilder$NullInputStream 0x18889aff "java.lang.ProcessBuilder$NullInputStream@18889aff"], :prev nil, :cmd ["git" "clone" "https://github.com/CFiggers/flutter-animate-transition"]}

Much simpler.

---

[1] You apparently need to know some commit sha or branch name from some other means. There doesn't appear to be a way to get that kind of info via the API.

There is a tags function for retrieving tags but if the repository doesn't have any, one gets a NullPointerException...

[sogaiu]

I don't know yet if the construct is valid but I found something our grammar doesn't seem to handle:

^#/(w/State ~fx-class #_m/SnackBar)
(...)

Assuming for the moment that that is valid, I presume the ^ indicates metadata. IIUC, #/(w/State ~fx-class #_m/SnackBar) is a tagged literal. So there is some metadata which is on (...) (the ... is meant to stand in for more code here).

I guess that's not a case I anticipated: https://github.com/sogaiu/tree-sitter-clojure/blob/421546c2547c74d1d9a0d8c296c412071d37e7ca/grammar.js#L335-L342

If we're lucky a fix might be a matter of attending to that last choice there somehow.

Adding something in the last choice there (and I guess may be in old_meta_lit) is one option.

This brings up the topic of whether that choice should be there restricting things...possibly it could be replaced with $._form.

I guess testing these possibilities out might be a next step to consider, though establishing the validity of the code might be a good idea too :)

[dannyfreeman]

Sounds like updating the file list should depend on us making sure we work with clojure dart properly first. I've commented over there with some thoughts

[sogaiu]

Below are some numbers about how many of each type of file extension I was able to find recently in "release" jars from Clojars:

.cljd 0
.clje 0
 .nbb 0
.cljr 2
  .bb 2
 .dtm 4
.boot 29
.cljx 175
 .edn 7441
.cljc 14418
.cljs 44140
 .clj 142449

It would be interesting to see if @phronmophobic has done any similar tallying of things collected for dewey.

For example, I expect that GitHub repositories would be more likely to have .bb files than Clojars.

[phronmophobic]

@sogaiu , not sure exactly what info you're looking for. I did a tally using the suffixes from your list and got the following results:

{".cljc" 14685,
 ".dtm" 21,
 ".clje" 28,
 ".cljx" 281,
 ".cljd" 29,
 ".cljs" 53838,
 ".bb" 117,
 ".boot" 778,
 ".clj" 178790,
 ".edn" 17424}

The local data on my laptop is a few months out of date, but hopefully should be useful as an estimate. If there's other info you're interested in, I can try to look into it. FWIW, all of the clojure repos that dewey tracks on github adds up to only about 15gb of compressed data.

[sogaiu]

@phronmophobic Thanks a lot for the summary and the numbers.

To give some background, the quality of tree-sitter-clojure relies heavily on testing against a large sample of source as partly a proxy for there not being an official specification (we've found constructs / usages via the testing that we hadn't imagined) but also to act as a check when we change things.

When I first started testing against source samples written by others I used git repository content. I believe at the time (this was a few years ago so my recollection may be untrustworthy) I was finding that there were too many cases where source content was off in one way or another (e.g. a missing delimiter or not realizing that what's inside comment can't be arbitrary text). It was also easier not to have to figure out which repositories had some kind of Clojure code in them -- probably something you are way more familiar with than me :)

I switched to using "release" jar content from Clojars figuring that the content was more likely to actually run (and hence ought to read / parse). Mostly that seems to have worked out.

However, I was only looking at .clj, .cljc, and .cljs files, which though they cover a fair bit of what files contain "Clojure" code, aren't the only things.

Recently we discovered that .cljd files tend to use a construct that doesn't appear to be used (much if at all) elsewhere and that this was not handled by our grammar. (FWIW, these files we found via GitHub repositories.)

That suggested to me that there might be files with other extensions that would be worth looking at. I don't imagine the list of file extensions I wrote about above misses a whole lot [1], but if there are others I'd like to investigate.

It looks like dewey has more files in total from your map as well as the 15gb compressed data number -- though may be that includes .git content? (For comparison, the last time I fetched jars from Clojars, the total size compressed came to a bit over 5gb -- though this includes .class files, .java files, etc.)

There's also the matter of possible bias -- possibly git repositories might be somehow more representative. So may be it's time to try out the grammar on the data identified by dewey :)

In any case, thanks again for sharing your results and the ongoing efforts with dewey.

---

[1] The list is basically a combination of what Cider and Helix use.

[phronmophobic]

though may be that includes .git content? 15gb is about the size of downloading a zip file for each of the latest commit on the default branch for each repo found. That means anything else tracked by git is included.

Dewey relies on github's notion of which libraries are "clojure" libraries. There are definitely some false positives and false negatives.

The most recent work on dewey includes automating the full pipeline so that dewey releases happen regularly (and cheaply) without any manual steps (except for clicking the release button at the very end). Part of the reason for this is to make it easier to support running more analyses. Depending on what you're looking for, I would be happy to chat about how to use dewey's data or do some ad-hoc analysis using dewey's data pipeline. If there's a simple way to try to parse or benchmark parsing across repos, then I could look into plugging that in.

[sogaiu]

Thanks for the further details and offer.

This bit of code shows what I'm working on putting in place.

Steps are roughly:

1. There's a directory tree that is traversed to collect file paths with certain extensions.
2. The collected file paths are written to a file.
3. The tree-sitter cli's parse subcommand is executed, being passed the path to the file with the collected file paths from the previous step.
4. Output from the command has to do with parsing issues and this is examined and portions saved to a file.

The output is a starting point for further investigation. In general I have found it necessary to examine the actual files that have had parsing issues as the tree-sitter cli output does not provide enough detail.

If there are more than a small number of cases, I've used clj-kondo linting results to try to create groups of files to try to make examining the results manageable.

These last "post-parsing examination" bits makes me wonder if it's a good fit for trying to use the data pipeline you've constructed.

The files I've had the least amount of experience examining are those that don't have the file extensions .clj, .cljc, or .cljs. This is repeating info from above, but for the sake of clarity, that includes:

• .bb
• .boot
• .cljd
• .clje
• .cljr
• .cljx
• .dtm
• .edn
• .nbb

If .clj, .cljc, .cljs, and .edn are put aside for the moment, it seems like the total number of repositories that have files with the remaining extensions would be under 1500.

Is it straight-forward to determine which repositories those might be?

I looked at all-repos.edn a bit and could see that repository urls seemed to be there, but I counted a bit over 14,000 of them so was a bit reluctant to try to get them all (at least in a short period of time) :)

On a side note, is "default" branch info for each repository saved somewhere? With that info, it seems like it might be possible to create urls to the .zip files (or a subset of them) you mentioned above.

[phronmophobic]

Is it straight-forward to determine which repositories those might be?

Yes. I have all the repos stored locally (which is how I found the tallies). When I get a free moment, I can create a list of repos and paths for the uncommon file suffixes.

On a side note, is "default" branch info for each repository saved somewhere? With that info, it seems like it might be possible to create urls to the .zip files (or a subset of them) you mentioned above.

Yes, that info can be found in the default-branches.edn.gz release file. Here's what the code that does the downloading looks like, https://github.com/phronmophobic/dewey/blob/main/src/com/phronemophobic/dewey/index.clj#L133.

I looked at all-repos.edn a bit and could see that repository urls seemed to be there, but I counted a bit over 14,000 of them so was a bit reluctant to try to get them all (at least in a short period of time) :)

Yea, it takes a while (a few hours?). Especially if you try to adhere to github's rate-limiting (which I do). There's no explicit rate-limit for non-API requests, but I try to keep it below their API request limit (5,000 requests an hour).

[phronmophobic]

Ok, here's a zip of paths for the less common suffixes and the github repos they were found in: paths.edn.zip

[sogaiu]

Wow, thanks so much for the zip file and the explanations!

[sogaiu]

After donig a bit of flitering, I ended up with 600 or so urls for .zip files: github-zips.txt

[phronmophobic]

You can also download individual files if you know the user, repo, and path. https://github.com/phronmophobic/dewey/blob/main/src/com/phronemophobic/dewey.clj#L167

[sogaiu]

Indeed.

Thanks for the tip!

[sogaiu]

Of the 1228 files that matched the file extension criteria, 7 produced parse errors.

The tldr is that each of the files had at least one issue to prevent successful parsing. I think none of them should have parsed correctly. It looks like we didn't find anything unexpected among the less common file extension file content...this time. But now we have code to repeat this sort of exercise in the future :)

---

Below are some descriptions about why parsing failed.

Content that has naked prose (non-Clojure code) and Clojure code in it

• scratch.edn

Content that looks like something that is being used as a placeholder / reminder for porting from some non-Clojure language to Clojure

• clojure.lang.Numbers%20TO%20PORT.cljx

Unintenionally malformed content (delimiter issue)

• shadow-cljs.edn

Template content that isn't strictly speaking Clojure, but quite close (uses file extensions often used by files that have "Clojure" in them [1])

• vase_service.edn
• vase_service.edn
• vase_service.edn

Intentionally malformed content

• malformed.edn

---

[1] Perhaps an unfortunate fairly prevalent practice from the perspective of folks doing these kinds of analyses :)

[sogaiu]

A note for future reference...

As seen earlier, clojure-mode.el listed build.boot and profile.boot. Apparently, one is not restricted to only file extensions in the context of package.json's file-types array: https://github.com/tree-sitter/tree-sitter/discussions/1083

So perhaps build.boot / profile.boot could be candidates too.

[sogaiu]

Since folks seem to keep coming up with additoinal Clojure flavors, it seems possible the file types may never be complete...

I'm going to close this issue for now. If there are particular file types that should be considered for addition, I suggest we make individual issues for them (ofc, multiple file types can be mentioned in a single issue :) ).

[sogaiu]

For future reference, Pulsar currently has this list:

  'boot'
  'clj'
  'clje'
  'cljr'
  'clj.hl'
  'cljc'
  'cljs'
  'cljs.hl'
  'cljx'
  'clojure'
  'edn'
  'bb'
  'joke'
  'joker'
  'jank'