Silently failed to send my statistics for a week

[kapooostin]

I haven't visited dashboard for a week or so. And just today I've found out that it is empty.

Statusbar shows that WakaTime OK. Sessions table in .wakatime.db is empty. Though heartbeats table is not.

WakaTime log contains the same repeating error for week:

{
  "now": "2019/04/30 14:01:39 +0300",
  "version": "10.8.2",
  "plugin": "sublime/3208 sublime-wakatime/8.3.4",
  "time": 1556622067.50238,
  "caller": "/Users/***/Library/Application Support/Sublime Text 3/Packages/WakaTime/packages/wakatime/api.py",
  "lineno": 140,
  "file": "/Users/***/Library/Application Support/Sublime Text 3/Packages/User/WakaTime.sublime-settings",
  "level": "WARNING",
  "message": "{'IOError': u'Could not find a suitable TLS CA certificate bundle, invalid path: /Users/***/Applications/NextGIS/Library/Frameworks/openssl.framework/Resources/ssl/certs/cert.pem', 'traceback': 'Traceback (most recent call last):\\n  File \"/Users/***/Library/Application Support/Sublime Text 3/Packages/WakaTime/packages/wakatime/api.py\", line 111, in send_heartbeats\
verify=ssl_verify)\\n  File \"/Users/***/Library/Application Support/Sublime Text 3/Packages/WakaTime/packages/wakatime/packages/requests/sessions.py\", line 555, in post\
return self.request(\\'POST\\', url, data=data, json=json, **kwargs)\\n  File \"/Users/***/Library/Application Support/Sublime Text 3/Packages/WakaTime/packages/wakatime/packages/requests/sessions.py\", line 508, in request\
resp = self.send(prep, **send_kwargs)\\n  File \"/Users/***/Library/Application Support/Sublime Text 3/Packages/WakaTime/packages/wakatime/packages/requests/sessions.py\", line 618, in send\
r = adapter.send(request, **kwargs)\\n  File \"/Users/***/Library/Application Support/Sublime Text 3/Packages/WakaTime/packages/wakatime/packages/requests/adapters.py\", line 407, in send\
self.cert_verify(conn, request.url, verify, cert)\\n  File \"/Users/***/Library/Application Support/Sublime Text 3/Packages/WakaTime/packages/wakatime/packages/requests/adapters.py\", line 226, in cert_verify\
\"invalid path: {0}\".format(cert_loc))\\nIOError: Could not find a suitable TLS CA certificate bundle, invalid path: /Users/***/Applications/NextGIS/Library/Frameworks/openssl.framework/Resources/ssl/certs/cert.pem\\n'}"
}

I can trace the reason: I deleted Next GIS just around this time. I have no idea why a certificate from the innards of Next GIS was used by WakaTime, but I guess it has something to do with the configuration of my machine.

Why WakaTime was OK all this time? And how do I fix this?

[kapooostin]

I tried to restart my machine, and now WakaTime is endlessly initializing.

[alanhamlett]

The good thing is, as long as you don't delete your .wakatime.db file your heartbeats should be saved there and synced to your dashboard once the plugin is fixed. This is because the warning message means the exception was caught and suppressed after saving an offline heartbeat. We show OK in this case because the plugin can't detect an intermittent network error from a recurring one, and periodically showing Error in the status bar when coding offline isn't a good experience.

The fastest way to fix this is to install Python3 from here: https://www.python.org/downloads/

Make sure to install in the default location, then WakaTime will detect and use this Python which comes with a suitable TLS CA cert for connecting to WakaTime servers.

[kapooostin]

I have python 3 installed. I also just linked it in WakaTime settings. It finds it alright and tries to send data but wakatime-core exited with status: 102.

[kapooostin]

Does it matter if I use Homebrew to install it or Python's own installer?

[alanhamlett]

Does it matter if I use Homebrew to install it or Python's own installer?

Homebrew is good, sorry I thought you were on Windows not OSX.

[kapooostin]

Python 3.6.8

[alanhamlett]

It finds it alright and tries to send data but wakatime-core exited with status: 102

That's the exit code corresponding to an API error, which is caused by the original error: IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /Users/***/Applications/NextGIS/Library/Frameworks/openssl.framework/Resources/ssl/certs/cert.pem

Does this folder exist? /Users/***/Applications/NextGIS/

I think when installing NextGIS it changed the default python for your system, and uninstalling broke it. What's the path of your python binary, for ex which pythonw or which python3.

[kapooostin]

Does this folder exist? /Users/***/Applications/NextGIS/

Nope, I deleted it.

➜ which python3
/usr/local/bin/python3

[alanhamlett]

I just upgraded the plugin with a new version of certifi which is the library that detects the location of ca cert files for wakatime. If you upgrade wakatime for Sublime, it might start working because the new version uses a bundled cacert.pem file.

[kapooostin]

Thank you! I guess it takes some time to get through Package Control.

Meanwhile I added REQUESTS_CA_BUNDLE to both .bash_profile and .zshrc and installed certifi. Strange things continue to happen. pip works fine in iTerm but MacOS Terminal never gets to verify certificate.

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),)': /simple/certifi/

[kapooostin]

I upgraded plugin up to 8.3.5 but the error didn't change:

OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /Users/***/Applications/NextGIS/Library/Frameworks/openssl.framework/Resources/ssl/certs/cert.pem

[alanhamlett]

Just released another version with a patched certifi library that forces using local package .pem file. Can you update and see if it works?

https://github.com/wakatime/wakatime/commit/8db46bb07dc3fd9e85480c3d05911f545f435475 was the fix, because the system's certifi library must be installed in the deleted NextGIS folder.

[kapooostin]

OK, I'll try it now.

For now I installed NextGIS again (and it fixed WakaTime immediately), and then checked its installation log. Finally I know what's going on. NextGIS uses setenv.*.plist files to setup ENV variables, that are left behind after its uninstallation. These two were the culprits:

/Users/<user>/Library/LaunchAgents/setenv.CURL_CA_BUNDLE.plist
/Users/<user>/Library/LaunchAgents/setenv.SSL_CERT_FILE.plist

[kapooostin]

Well, it works when I put a wrong URI in both setenv.CURL_CA_BUNDLE.plist and setenv.SSL_CERT_FILE.plist. And it works when I remove them completely. I guess the issue is solved?

[alanhamlett]

I guess the issue is solved?

Yes, glad it's working. Just curious, does it also work with v8.3.6 and the /Users/<user>/Library/LaunchAgents/setenv.CURL_CA_BUNDLE.plist present but NextGIS app removed?

[kapooostin]

Yes, it works. But I cannot be sure what exactly solved this issue, because pip in Terminal also stopped showing any warnings after I installed and deleted NextGIS once again. Or could your change also help that?

[alanhamlett]

An easy way to tell is run this command: sudo mv /Users/<user>/Applications/NextGIS /Users/<user>/Applications/NextGIS-bak

Then if pip in Terminal breaks but WakaTime works we know the latest update fixed it. That update forces using a locally bundled ca bundle file which should bypass the plist configs.

[kapooostin]

I deleted NextGIS immediately after I installed it again (but saved its cert.pem just in case though moved it to another folder). And both Terminal and pip work ever since.

WakaTime updated silently so when I tried to update it after your letter, it was already 8.3.6. I guess, to check it properly I need to reinstall and delete version of NextGIS that I had before, and downgrade WakaTime.

I could try it if you need this test.

[alanhamlett]

I could try it if you need this test

No that's fine, as long as it's working for you I don't want to take up more of your time. I can reproduce this on my end if needed.