search

wakatime / wakatime-mode

Emacs plugin for automatic time tracking and metrics generated from your programming activity.
https://wakatime.com/emacs
267 stars 41 forks source link

How does the wakatime-mode recognized the language? #37

Closed Liutos closed 2 years ago

Liutos commented 5 years ago

I found there're about 55% of my languages is "Other", why the wakatime-mode can't recognized the language I used. Actually I spent a lot of time in writing some .org files, does wakatime-mode not treat the Org as a language? Thanks~

sawthinkar commented 4 years ago

I have the same issue here. Is there a way to get Other languages recognized?

alanhamlett commented 4 years ago

There's two ways to add support for languages:

  1. A file extension rule server-side
  2. Add/update the corresponding Pygments Lexer for the language

Do org-mode files always end with .org?

sawthinkar commented 4 years ago

In my case is .story files which are for Storyscript.

Anton-Latukha commented 4 years ago

@alanhamlett

Yes, Org-mode files always use .org extensions.

Anton-Latukha commented 4 years ago

Solid explanation:

I've just tested this.

Yes, Org-mode files always use .org, since they can embody a lot of other languages - and so file utility classifies it differently based on the beginning of the file & contents. I've got LaTeX, ASCII text Emacs Lisp etc, so the distinction for the file is its extension.

Emacs itself recognizes Org-mode files based on the extension, I've just tested and observed that.

Anton-Latukha commented 4 years ago

Other IDEs/editors also have support of Org-mode to some degree.

So the extension is the most efficient way overall.

Anton-Latukha commented 4 years ago

Also studies the usage of .org extension. We can conclude .org is sole for `Org-mode:

Source #1: https://filext.com/file-extension/ORG - some Hollywood FX by Pinnacle Systems video editing software extension. Source #2: https://fileinfo.com/extension/org - here we observe that .org in IT was used in the most ancient versions of Lotus Organizer https://en.wikipedia.org/wiki/IBM_Lotus_Organizer, somewhere in- between years 1992-1994.

SteamedFish commented 3 years ago

I'm quite confused because, wakatime cannot even recognize golang with Emacs. All golang becomes Other. I tried vscode, found that wakatime can recognize golang with vscode.

There must be some difference between wakatime emacs extension and wakatime vscode extension, which I haven't figure out.

There's two ways to add support for languages:

1. A file extension rule server-side

2. Add/update the corresponding [Pygments Lexer](https://github.com/pygments/pygments/tree/master/pygments/lexers) for the language

Do org-mode files always end with .org?

notarock commented 3 years ago

Bumping for interest - is there any way I can help make this plugin works with detecting the language for org-mode?

thomasf commented 3 years ago

the wakatime cli has a --language option, can that be used? Emacs could base the language on which mode is actually active (the major mode) instead of file extensions which is probably what you'd want. This would maybe not work with all modes that enables multiple major modes in one buffer but it probably should include a mode to file type table in wakatime-mode anyway and just resort to default wakatime functionality if there isn't a match.

Anton-Latukha commented 3 years ago

Above statement is not completely true.

The core design of Emacs is that Emacs has only one major-mode at one time. That is why there is a distinction between major-mode (main mode that is loaded) and additionally loaded minor-modes. There may be only one major-mode at a time. Other modes should be loaded as minor-modes. Mixing several major-modes are not officially supported, and considered hacking the Emacs system. There is a packages that hack the system by creating custom major-mode and importing several major-modes into it, but that is considered direct hacking and refuted by Emacs maintainers and support.

So there is only one major-mode at a time. If people hack-in two in there - they are the ones that should care to make that hack work.

The name of the current major-mode Emacs operates in is stored in major-mode variable.

So, it is enough to bind the list of the major modes to the languages and tasks.

And even simplier and automatic:

The modes are direcrly named after the languages.

org-mode
markdown-mode
elisp-mode
haskell-mode

So just dropping the -mode and passing that string into wakatime should be enough to support most cases by one action.

thomasf commented 3 years ago

The core design of Emacs is that Emacs has only one major-mode at one time.

Well you do have all the stuff that supports multiple major modes for one buffer. Some of them uses multiple backing buffers with one major mode per buffer and indirect buffers to piece a buffer together and I am not completely sure that major-mode variable will always be correct in those cases. I could be wrong though.

https://www.emacswiki.org/emacs/MultipleModes

I think even org-mode supports this for babel code blocks these days but I might be misremembering that. Earlier editing org babel blocks in the language mode requires creating an indirect buffer for that block and edit in.

So just dropping the -mode and passing that string into wakatime should be enough to support most cases by one action.

I think it's better to use an explicit list or leave the unknown modes to wakatime to figure out itself.

thomasf commented 3 years ago

Mixing several major-modes are not officially supported, and considered hacking the Emacs system.

What does this even mean? Emacs is supposed to be flexible and the hackability factor is a part of the ethos of the whole project

Anton-Latukha commented 3 years ago

Well. @thomasf.

Person is asked to suggest a good solution. I pointed out the official way.

And we are trying to find a good solution here. And I shown the most simple, effective, and legitimate official way to support all languages and particular modes of operation for the wakatime, because Emacs with the major-mode literally indicates the purpose of the active tab the person works in.

You not propose the solution while trying to refute the official one. That is not helping.

Answering:

Mixing several major-modes are not officially supported, and considered hacking the Emacs system.

What does this even mean? Emacs is supposed to be flexible and the hackability factor is a part of the ethos of the whole project

That Emacs Lisp is Turing complete does not mean that everything you write in it is a perfect clean code and valid and that Emacs does not have any design.

With this type of argument, "Since language is Turing complete, and the program allows extending it, so I can do everything, so you should expect everything" - we can say that everything is impossible, since VSCode written in Turing complete language, so we should expect everything (full chaos) from the system, which is impossible to prepare for, and so for VSCode and Emacs there is no design and so also no solution to any task. Ex falso sequitur quodlibet

Programs have a particular design. They are written in a particular structured way, and impose design and interfaces on the extensions. Emacs does the same. Emacs is extensible, it also has a design, official support, official deprecations and restriction, and good practices and proper way of doing things.

In the debate of "What is Emacs", you do not choose what to pick as the answer, sorry, but what is Emacs - Emacs determines.

When one glances over, ignores and actively passes by so much red flags shown to him on the runway by everyone - there is no need to tell a person that he is responsible for the decision to mix-mash major-modes into one unsupported monstrosity, when it is clearly put that Emacs does not support it, and creators of modes and those modes themselves at all not expect and are not developed to suddenly mix-mesh them together into one, no one knows what gets overshadowed by what when you mix random modes together, particularly how about that then the official standard operation key binds of modes clash, what C-c C-c does is in your case of modes mix-mash?

StackOverflow "Emacs: Could one buffer have more than one major mode?": https://stackoverflow.com/questions/31736638/emacs-could-one-buffer-have-more-than-one-major-mode

No, but...

Official Emacs mailing list: https://lists.gnu.org/archive/html/emacs-devel/2016-04/msg00677.html

The changes that will be needed for supporting multiple modes in the same buffer will be extensive, whether you want it or not, so trying too hard to make it easier on modes to adapt will skew the design.

Wikipedia, article "Emacs": https://en.wikipedia.org/wiki/GNU_Emacs#Minor_modes

A GNU Emacs editing buffer can use only one major mode at a time, but multiple minor modes can operate simultaneously. There is also a minor mode that allows multiple major modes to be used in a single file, for convenience when editing a document in which multiple programming languages are embedded.

Emacs Wiki:

What a shame that there are multiple multiple mode modes. As soon as you need mode X that requires multiple mode M1, while already using another mode which uses multiple mode M2, you’re loading far more than you need to. Just because TIMTOWTDI (sorry for the perl-ism) doesn’t mean you must do it more than one way.

And at last, the main official source, fist link when one web searches "emacs major mode": GNU.org, Emacs, article "Major-Modes": https://www.gnu.org/software/emacs/manual/html_node/emacs/Major-Modes.html

Every buffer possesses a major mode, which determines the editing behavior of Emacs while that buffer is current. Since every buffer has exactly one major mode, there is no way to “turn off” a major mode; instead, you must switch to a different one.

Anton-Latukha commented 3 years ago

Also by sheer coincidence, I work right under the main Emacs maintainer, if it is still not clear that Emacs has one major-mode for a tab ("buffer" in Emacs terms), I can go on a limb and ask official response just for us one more time, but it would be a pity to bother a guy for that.

Person writing this happens to write the: https://github.com/Anton-Latukha/Fundamental-Haskell

And I may say, when I write that, I write in the Org mode markup. No. Not in LaTeX math formulas. not in Haskell, it is in Org mode. That there are the snippets of code inside there - is not relevant. It is a book written in Org-mode. The major work on it and the modus operandi is the Org mode. One opens the Org-mode, because one wants to put code snippets particularly, particularly into the Org-mode markup.

Why participating in the thread? - I am happy for wakatime at the end of the week to show a number of hours I've spent writing notes and the book this week, What is inside the feature of Org-mode called Org-Babel blocks - is a petty detail that should be looked over for simplicity and effectiveness. Otherwise, you just try to overcomplicate things for people that provide the Open Source project to you for free, meanwhile listening to your complaints while trying to help you.

Anton-Latukha commented 3 years ago

The official Emacs solution is simple:

  1. Emacs has one major-mode at a time. It is particularly done so. for us (here), so we can get info of what is the current main activity:

    Every buffer has exactly one major mode ... Every buffer possesses a major mode, which determines the editing behavior of Emacs while that buffer is current.

  2. Name of it is stored in the major-mode variable.
  3. Value of the variable is <...>-mode where <...> being in the big majority of cases purposefully and conveniently being literally the name of the language, always in lowercase. wakatime can be literal and show the Emacs user the direct <...> part of the name of the mode, aka org-agenda, it is even better doing so this literal way, since person receives the true data he understands, the org is note taking activity, and org-agenda is scheduling and time management, it is true for all other Emacs mode cases, just show Emacs'ers the major-mode name - and they would hag you and be marveled.
thomasf commented 3 years ago

examples

You can most likely find a lot of more examples if you just spend some time on looking...

There definitely has to be a fallback to a list that is a defcustom and support some kind of matching between mode names and source types with the option to defer to wakatimecli/pygments because they will not always match directly by stripping a -mode suffix from the major-mode name. The defcustom can come with a set of defaults that are maintained inside this package as a default.

thomasf commented 3 years ago

Also by sheer coincidence, I work right under the main Emacs maintainer, if it is still not clear that Emacs has one major-mode for a tab ("buffer" in Emacs terms), I can go on a limb and ask official response just for us one more time, but it would be a pity to bother a guy for that.

I have written at least tens of thousands of lines of emacs lisp over the years so I know the language and environment very well thank you., stop trying to make this into some weird pissing contest.

Otherwise, you just try to overcomplicate things for people that provide the Open Source project to you for free, meanwhile listening to your complaints while trying to help you.

Well, for me wakatime cli is actually already handling this perfectly fine by its own so that simple solution is to add no new code to handle this. It seems like some people some how have a broken wakatime cli install, maybe old pygments (?).

thomasf commented 3 years ago

I just wrote from a practical perspective..

Those multi major modes exists and they are in use so regardless how official the support is we should not create code in this package that might break or do the wrong thing if those packages are in use.

There is no reason to break compatibility just because one's own idealised opinion how how something should work when it actually isn't guaranteed to work like that in the real world.

Anton-Latukha commented 3 years ago

@thomasf Answering to:

* js-mode, js2-mode and js3-mode exists and are all major modes

* enh-ruby-mode (enhanced ruby mode)

* there are github and gitlab flavoured markdown-modes with different major mode names,  I think these can change the major mode depending on which source block is active so the same file will report as different languages depending on where in the file the cursor is when wakatime cli is invoked.. This might or might not be seen as an intended consequence.

* web-mode supports both javascript and html so it's mode name does not even reveal which of the file types it is so in this case I would just defer detection to pygments as the easy solution.  I for sure don't want my html editing to be categorizes as a "web" file type.

--language LANGUAGE accepts only valid values, and js{,2,3} - and others you list at are not valid language values, they are just ignored.

If you read:

wakatime --help

...

  --language LANGUAGE   Optional language name. If valid, takes priority over
    auto-detected language.

We are in this thread because default wakatime autodetection does not work for the swath of languages in Emacs.

Citing you "I just wrote (this) from a practical perspective". Very simple code. Take a variable value, drop the -mode from it, and send that info to wakatime.

And with PR you trying to block - if the value matches valid language - language is detected. If value is not valid - the current behavior (default auto-detection) launches.

This way it already works way better.

Anton-Latukha commented 3 years ago

Responding: https://github.com/wakatime/wakatime-mode/issues/37#issuecomment-702195992

I just wrote from a practical perspective..

Those multi major modes exists and they are in use so regardless how official the support is we should not create code in this package that might break or do the wrong thing if those packages are in use.

Sorry, multiple major modes are not supported by the GNU Emacs. Do not try to block the development of Emacs just because you broke the rules of the design of Emacs.

There is no reason to break compatibility just because one's own idealised opinion how how something should work when it actually isn't guaranteed to work like that in the real world.

That is not mine opinion, it is an opinion of the GNU Emacs project. While your opinion is that you expect that your hacks should be supported.

For example - I am not an idealist - I do not expect my hacks to systems to be supported by all others.

Compatibily can not be broken, because there is no protocol of compatibility established and provided by you to Emacs that they should maintain. While you broke the protocol of Emacs compatibility.

Anton-Latukha commented 3 years ago

And over that - the names of unsupported multiple major-modes that you have, would not be recognized by wakatime as valid language names, since you are the one that named them and they should have a different name to the modes you mixed, so you should not be worried, the code submitted would work for you just as before.

thomasf commented 3 years ago

You could easily have implemented better handling which is easy to implement and not unreasonable instead of arguing with me about not doing it in the same amount of time... I'm blocking you now so I won't have to get any further notifications.

Anton-Latukha commented 3 years ago

There was no talk about development. You right of the gate simply blocked the PR without reading and understanding how it works, without any review, there were no suggestions on how to improve the code.

What was send out is a demand to not accept it and complain that code is not good enough because it is not a not existent ephemeral ideal code you imagine.

I've could extend the code, but so much already works great for everybody, there is no need to make defcustoms and extending it, since it is already resilient.

Ыo Emacs right away sends and wakatime accepts the language setting and so we rely less on heuristic autodetection which does not detect some stuff, this report exists because autodetection does not detect some stuff. So now we would use the both ways of detection simultaneously, and BTW to just send the major-mode info is way faster and also fewer computations.

thomasf commented 3 years ago

There was no talk about development. You right of the gate simply blocked the PR without reading and understanding how it works, without any review, there were no suggestions on how to improve the code.

quoute from one of my earlier comments in this thread:

"There definitely has to be a fallback to a list that is a defcustom and support some kind of matching between mode names and source types with the option to defer to wakatimecli/pygments because they will not always match directly by stripping a -mode suffix from the major-mode name. The defcustom can come with a set of defaults that are maintained inside this package as a default."

maybe something along the lines of this which makes the behavior user defined and can be turned off completley (setting custom value to nil) for those who prefer not overriding wakatimes own identification method. 

(defun wakatime-guess-file-type-from-major-mode ()
  (downcase (string-trim-right (symbol-name major-mode) "-mode")))

(defcustom wakatime-language-finder
  '((js2-mode "javascript") ;; literal mode to language
    (js3-mode "javascript") 
    (some-mode t) ;; defer to wakatime-cli
    (t wakatime-guess-file-type-from-major-mode) ;; if function returns nil, matchers continues to the next line, if t defer to wakatime-cli, if function returns string it is used as language.
)
  "...")

If you thought it was unclear what I was talking about you could just have asked me to clarify instead of trying to justify that your solution is the only valid way forward.

What was send out is a demand to not accept it and complain that code is not good enough because it is not a not existent ephemeral ideal code you imagine.

I made a recommendation which most people would see as less strong than a demand. It's also a recommendation that is up to the maintainer/repository owner to do decide on, I just give my opinion. I also said that I don't recommend the patch in it's current form, if it's improved I have no issues with it.

thomasf commented 3 years ago

The core design of Emacs is that Emacs has only one major-mode at one time.

Yes and those multiple major modes modes technically uses various methods to change the major mode depending on the context like where in the buffer the cursor is so they are not modifying Emacs internals to achieve their goals.

This might or might not be what a user of wakatime-mode wants so it should be possible to opt in or out of that behavior.

Try setting markdown-fontify-code-blocks-natively to t in markdown-mode's (on melpa) gfm-mode variant and create this markdown buffer (without spacing in between backticks, obviously)

` ` ` elisp
try c-h v major-mode when the cursor is in this block.
Tt will say emacs-lisp-mode even if the container file
itself is github flavoured markdown.
` ` `

This is a real world example of a mode that many people obviously are using (1,383,373 downloads from melpa, 16th most downloaded package) where a simple major-mode matching might not want be what a user of wakatime-mode wants.

Similar issues are probably present in all cases where one of the multi major mode modes are in use.

With this knowledge in mind we now know that straight up merging your pull request will break usage for a subset of markdown-mode users if they use gfm-mode and native block fontification enabled..

This is not about made up examples or invalid use of major modes at all. Please lets focus on adding mechanisms so that users of wakatime-mode can avoid that if they don't want it before any pull request is merged.

Anton-Latukha commented 3 years ago

Please, understand how the recievement and prioritization of information works in wakatime.

Example still does not apply.

markdown-mode would still return markdown as a language name, which would get accepted by wakatime --language as a viable language value, and therefore prioritized.

And stop blocking work.

thomasf commented 3 years ago

Thats good then.

I still prefer a simple customization to opt out of this feature. For me wakatime-cli's own detection works fine and I don't want to override it.

And again , I am not blocking. I am not a maintainer and I can only make suggestions and recommendations. I have no power to block anything.

Anton-Latukha commented 3 years ago

If I understood your example - you show that what you wanted from modes - to respect the code block language - indeed would work with the proposed major-mode process.

If you write a elisp code in the block - it would count it as ELisp code writing - which is great.

You actually provided a point in favour of using this major-mode process.

thomasf commented 3 years ago

If you write a elisp code in the block - it would count it as ELisp code wtiring - wich is great

I would fork wakatime-mode if it started doing this without an opt out. I don't want the same file to report as different language types. I need to know specifically how much time I spend on documentation.

Anton-Latukha commented 3 years ago

I am happy, it is great that is works for you and markdown, but official Org-mode Emacs documentation does not work in wakatime at all currently, and I submitted work to support it.

I submitted patches to wakatime also, so it indeed would do work: https://github.com/wakatime/wakatime/pull/233. And this code submitted to wakatime-mode would allow to support org and many other languages for people.

After code is merged - you always can improve it however you want it, by sending patches.

Of cource you are also allowed to fork the project.

Anton-Latukha commented 3 years ago

wakatime has an option for languages, called --language.

When you write Markdown - the language is Markdown, when you open a elisp block and write Emacs Lisp code - it is Emacs Lisp language. Emacs reports the language into wakatime --language option.

If you want --language to be an "activity" tracking option - sorry - you mixed-up its use - it is not activity tracking - --language option is obviously for programming language reporting.

  --language LANGUAGE   Optional language name. If valid, takes priority over
                        auto-detected language.

For activity there is:

  --category CATEGORY   Category of this heartbeat activity. Can be "coding",
                        "building", "indexing", "debugging", "running tests",
                        "writing tests", "manual testing", "code reviewing",
                        "browsing", or "designing". Defaults to "coding".

If you want "writing notes/documentation" to be tracked - you need to upstream that option into wakatime upstream.

thomasf commented 3 years ago

What is Org-mode Emacs documentation?

Is it org mode files without an .org extension?

I just added an .org file to a project I am working on right now and it got identified correctly at once (last entry in the list)

image

Also my notes repository which is almost all org mode files shows org mode as the language everywhere..

Anton-Latukha commented 3 years ago

Well - it works for now because I upstreamed the patch into wakatime itself, I am happy to see that it works now.

thomasf commented 3 years ago

I don't belive so but I could be wrong.. wakatime --version says 13.0.7 which is the version that was releases in february so I'm not sure your linked patch mentioned above that was merged 12 days ago is not yet released.

Anton-Latukha commented 3 years ago

May be.

It still not arrived into my system, and I not looked completely into automatic resolving of languages, in my case, in years of use - wakatime it never recognized the org, and I tracked the reason down to this thread.

The reason of this report opened, its heading post and down the thread report that org does not get recognized automatically: https://github.com/wakatime/wakatime-mode/issues/37#issue-383351185

thomasf commented 3 years ago

I found the reason..

I have added it under custom rules in the wakatime web ui settings.

image

alanhamlett commented 2 years ago

It's 2022! There've been some changes around language detection:

For now, I'll close this. If the new wakatime-cli isn't detecting .org files then create an issue and we'll add support for that and any language.