wakatime / vscode-wakatime

Visual Studio Code plugin for automatic time tracking and metrics generated from your programming activity.
https://wakatime.com/vs-code
BSD 3-Clause "New" or "Revised" License
1.21k stars 134 forks source link

Remote mode doesn't work: `filter file: skipping because of non-existing file` #314

Closed clouedoc closed 1 year ago

clouedoc commented 1 year ago

Hi, I cannot send heartbeats when editing on an SSH remote.

I get this suspicous message in .wakatime.log: filter file: skipping because of non-existing file I also get the following: no heartbeats left after filtering. abort heartbeat handling.

Here are log lines from a recent programming session on an SSH Remote:

{"caller":"cmd/heartbeat/heartbeat.go:72","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"heartbeat.SendHeartbeats","level":"debug","lineno":45,"message":"params: api params: (api key: '<hidden>e6db', api url: 'https://api.wakatime.com/api/v1', backoff at: '', backoff retries: 0, hostname: 'iCamillou.local', key patterns: '[]', plugin: 'vscode/1.73.1 vscode-wakatime/22.0.1', proxy url: '', timeout: 2m0s, disable ssl verify: false, ssl cert filepath: ''), heartbeat params: (category: 'coding', cursor position: '10', entity: '/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts', entity type: 'file', num extra heartbeats: 0, is unsaved entity: false, is write: false, language: '', line number: '45', lines in file: '48', time: 1670001884.35757, filter params: (exclude: '[]', exclude unknown project: false, include: '[]', include only with project file: false), project params: (alternate: 'censored', branch alternate: '', disable submodule: '[]', map patterns: '[]', override: ''), sanitize params: (hide branch names: '[]', hide project folder: false, hide file names: '[]', hide project names: '[]', project path override: '/home/clouedoc/Code/censored')), offline params: (disabled: false, print max: 10, queue file: '', num sync max: 1000), status bar params: (hide categories: false)","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/heartbeat/format.go:18","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"heartbeat.WithFormatting","level":"debug","lineno":45,"message":"execute heartbeat filepath formatting","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/heartbeat/format.go:61","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"heartbeat.formatLinuxFilePath","level":"warning","lineno":45,"message":"failed to resolve real path for \"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts\": lstat /System/Volumes/Data/home/clouedoc: no such file or directory","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/heartbeat/entity_modify.go:15","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"heartbeat.WithEntityModifer","level":"debug","lineno":45,"message":"execute heartbeat entity modifier","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/filter/filter.go:26","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"filter.WithFiltering","level":"debug","lineno":45,"message":"execute heartbeat filtering","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/filter/filter.go:33","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"filter.WithFiltering","level":"error","lineno":45,"message":"filter file: skipping because of non-existing file \"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts\"","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/remote/remote.go:49","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"remote.WithDetection","level":"debug","lineno":45,"message":"execute remote file detection","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/apikey/apikey.go:31","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"apikey.WithReplacing","level":"debug","lineno":45,"message":"execute api key replacing","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/filestats/filestats.go:23","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"filestats.WithDetection","level":"debug","lineno":45,"message":"execute filestats detection","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/language/language.go:19","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"language.WithDetection","level":"debug","lineno":45,"message":"execute language detection","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/deps/deps.go:38","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"deps.WithDetection","level":"debug","lineno":45,"message":"execute dependency detection","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/project/filter.go:23","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"project.WithFiltering","level":"debug","lineno":45,"message":"execute project filtering","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/heartbeat/sanitize.go:30","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"heartbeat.WithSanitization","level":"debug","lineno":45,"message":"execute heartbeat sanitization","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/remote/remote.go:113","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"remote.WithCleanup","level":"debug","lineno":45,"message":"execute remote cleanup","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/filter/filter.go:52","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"filter.WithLengthValidator","level":"debug","lineno":45,"message":"no heartbeats left after filtering. abort heartbeat handling.","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"cmd/heartbeat/heartbeat.go:56","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"heartbeat.Run","level":"debug","lineno":45,"message":"successfully sent heartbeat(s)","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/offline/offline.go:106","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"offline.WithSync","level":"debug","lineno":45,"message":"execute offline sync with file /Users/clouedoc/.wakatime.bdb","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"pkg/offline/offline.go:146","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"offline.Sync","level":"debug","lineno":45,"message":"no queued heartbeats ready for sending","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}
{"caller":"cmd/offlinesync/offlinesync.go:48","file":"/home/clouedoc/Code/censored/src/packlets/account-gen/insert-account-db.ts","func":"offlinesync.Run","level":"debug","lineno":45,"message":"successfully synced offline activity","now":"2022-12-02T18:24:44+01:00","os/arch":"darwin/arm64","plugin":"vscode/1.73.1 vscode-wakatime/22.0.1","time":1670001884.35757,"version":"v1.60.1"}

Here is the CLI command that is called by the Wakatime VSCode extension: (seen in debug logs)

# When editing a file on my local computer (works)
/Users/clouedoc/.wakatime/wakatime-cli-darwin-arm64 --entity /Users/clouedoc/wakatimelogs --plugin "\"vscode/1.73.1 vscode-wakatime/22.0.1\"" --lineno 19 --cursorpos 1 --lines-in-file 19 --write

# When editing a file on an SSH remote (does not work)
/Users/clouedoc/.wakatime/wakatime-cli-darwin-arm64 --entity /home/clouedoc/Code/another-censored-project/.pre-commit-config.yaml --plugin "\"vscode/1.73.1 vscode-wakatime/22.0.1\"" --lineno 7 --cursorpos 1 --lines-in-file 7 --alternate-project my-censored-project --project-folder /home/clouedoc/Code/another-censored-project --write
clouedoc commented 1 year ago

New infos: the entity string should indicate that it is remote somehow. Or the file filter should be skipped.

From the wakatime-cli repository:

func filterFileEntity(h heartbeat.Heartbeat, config Config) error {
    if h.EntityType != heartbeat.FileType {
        return nil
    }

    if h.IsUnsavedEntity {
        return nil
    }

    if h.IsRemote() {
        return nil
    }

    entity := h.Entity
    if h.LocalFile != "" {
        entity = h.LocalFile
    }

    // skip files that don't exist on disk
    if _, err := os.Stat(entity); os.IsNotExist(err) {
        return fmt.Errorf(fmt.Sprintf("skipping because of non-existing file %q", entity))
    }
// IsRemote returns true when entity is a remote file.
func (h Heartbeat) IsRemote() bool {
    if h.EntityType != FileType {
        return false
    }

    if h.IsUnsavedEntity {
        return false
    }

    return remoteAddressRegex.MatchString(h.Entity)
}

Link It should match the following regular expression:

// remoteAddressRegex is a pattern for (ssh|sftp)://user:pass@host:port.
var remoteAddressRegex = regexp.MustCompile(`(?i)^((ssh|sftp)://)+(?P<credentials>[^:@]+(:([^:@])+)?@)?[^:]+(:\d+)?`)
clouedoc commented 1 year ago

I think that something could be done here to add SSH remote information to the heartbeat.

It could pass information about the remote host.

There is already relevant sanitization happening inside wakatime-cli:

func hideCredentials(h Heartbeat) Heartbeat {
    if !h.IsRemote() {
        return h
    }

    match := remoteAddressRegex.FindStringSubmatch(h.Entity)
    paramsMap := make(map[string]string)

    for i, name := range remoteAddressRegex.SubexpNames() {
        if i > 0 && i <= len(match) {
            paramsMap[name] = match[i]
        }
    }

    if creds, ok := paramsMap["credentials"]; ok {
        h.Entity = strings.ReplaceAll(h.Entity, creds, "")
    }

    return h
}

https://github.com/wakatime/wakatime-cli/blob/58c6156e178033f9d881a45dea24b5ec890b8b3c/pkg/heartbeat/sanitize.go#L107-L126

clouedoc commented 1 year ago

I'm unsure, but maybe relevant information can be extracted from uri instead of fileName: https://github.com/microsoft/vscode/blob/69c8a7594d5efbc391d873157184b0317991cd11/src/vs/workbench/api/common/extHostDocumentData.ts#L61-L63

alanhamlett commented 1 year ago

Are you running the wakatime extension installed on the remote vscode container? In that case, it should be running your two wakatime-cli commands on different machines. The first one # When editing a file on my local computer (works) would execute on your host machine, and the second one # When editing a file on an SSH remote (does not work) would execute on your remote machine. That means relative to your remote machine, /home/clouedoc/Code/another-censored-project/.pre-commit-config.yaml should exist. For ex: if you ssh into your remote machine and type ls -l /home/clouedoc/Code/another-censored-project/.pre-commit-config.yaml you should see the file's info printed.

clouedoc commented 1 year ago

Hello @alanhamlett, thanks for checking in!

I am running the extension in UI mode as explained in the README.md: https://github.com/wakatime/vscode-wakatime#ssh-configuration. Additionally, I made sure to remove all wakatime-related files and uninstalling the WakaTime extension on the server side.

It's not a container but rather a shared SSH server used by my coworker and I. Running the wakatime-cli command on the server will cause one of us to have additional undeserved heartbeats, so I had to to disable Wakatime and thus loose a good share of my stats.

If I'm correct, running the extension in UI-only mode causes the command to be issued locally even though I am editing a remote file. This then causes the file existence checks to fail since it does not exist on my laptop, but rather on the server.

This might be fixable by providing wakatime-cli with this syntax when editing a remote file: (ssh|sftp)://user:pass@host:port:/home/user/my-project/hello.md (instead of /home/user/my-project/hello.md). This is because the heartbeat will be marked as remote, thus exiting the check function early and bypassing the file existence check.

As far as I've searched I didn't find anything that would suggest that this is a problem related to my configuration but rather a total lack of feature by the extension, even though it sorts-of claims that it does support running in UI-only mode in the latest README.md. Other people might be experiencing the same problem (#304 - not sure about what protocol could be used to describe the file in this case, though...)

alanhamlett commented 1 year ago

If I'm correct, running the extension in UI-only mode causes the command to be issued locally even though I am editing a remote file.

Yes, correct.

I see the problem now, and we can get a new vscode-wakatime version released today to fix it for you. The Document.uri property has scheme: "vscode-remote" when running locally (ui) and connected to a remote SSH server.

alanhamlett commented 1 year ago

Fixed in v22.1.0 just released today.

clouedoc commented 1 year ago

@alanhamlett Thank you šŸ™

// TODO: how to support 'dev-container', 'attached-container', 'wsl', and 'codespaces' schemes?

To answer your question, here are some solutions I thought of:

  1. Support alternative schemes in wakatime-cli (it might be useful to some user to see how much time they spent in a devcontainer VS their local computer; for instance, to see how much they have containerized their development workflow)
  2. filename.replace(/(dev-container|attached-container|wsl|codespaces):\/\//g, "ssh://")
  3. Support a --skip-file-existence-check in wakatime-cli

Of course, you know better than me, it's just some food for thoughts. I don't care, as I am now a satisfied user šŸ™‚

alanhamlett commented 1 year ago

It might work already for wsl without changes, and so far most people seem to be using dev-container in a solo environment not shared so they're fine running the wakatime plugin in the remote container. Let's just wait and see if anyone reports an issue around WSL or Dev Containers.