JSON blob for "new e-mails"?

[EAFSC]

Is there like a simple https://tutanota.com/check-for-new-emails endpoint that I can query with a POST request containing my username/e-mail addy and password and which then spits back a simple JSON blob such as:

{
    "unread e-mails": 0,
    "last received timestamp": 124354353421,
    "total e-mails": 123
}

Or something like that? Since Tutanota doesn't support IMAP (unless you pay a bunch of money), it's a major PITA to constantly manually log in to check for new e-mails (only to find that nobody has replied). This would be so great to be able to automate at least such a "notification".

Note that I didn't ask up-front for a full-fledged API or "sneak-around" for the missing IMAP support; I don't expect you to let me actually list the e-mails in that JSON blob. If you are feeling extremely generous, however, maybe you would let us see the subjects of the e-mails at least? But even just this basic JSON blob would be great, and IMO a good compromise between the paid account with actual IMAP access and the free, limited ones.

[charlag]

There's an SSE endpoint which is used by Desktop client and Android client. You could check DesktopNotificator.

We cannot "just give you subjects" because they're encrypted.

[EAFSC]

Sorry... I couldn't understand this reply at all. DesktopNotificator? No such thing mentioned anywhere outside this issue on this repository, and all of Github only has something I don't even get what it is?

[snaggen]

https://github.com/tutao/tutanota/blob/master/src/desktop/DesktopNotifier.js

Perhaps this?

[EAFSC]

@snaggen Thanks, but I see no URL in there or anything?

[charlag]

Ah, sorry, I meant DesktopSseClient https://github.com/tutao/tutanota/blob/master/src/desktop/DesktopSseClient.js

URL is https://mail.tutanota.com/sse https://github.com/tutao/tutanota/blob/63996711170950704ee133a5e2b1ecbfdadc0dab/src/desktop/DesktopSseClient.js#L111

I'm not sure what do you want to achieve in the presence of mobile/desktop clients but I hope it will help. We plan to add background notifications to the web client eventually as well.

[charlag]

Also Android implementation: https://github.com/tutao/tutanota/blob/63996711170950704ee133a5e2b1ecbfdadc0dab/app-android/app/src/main/java/de/tutao/tutanota/push/PushNotificationService.java

[EAFSC]

I've been staring at the URLs you've posted for hours, and tried everything I could think of, but I just get these errors:

"HTTP method GET is not supported by this URL" "HTTP method POST is not supported by this URL"

I connect to https://mail.tutanota.com/sse with either GET or POST with one parameter: '_body', and it's set to the URL-encoded JSON version of this array:

$JSON_data = 
[
    '_format' =>        '0',
    'identifier' =>     '123',
    'userIds' =>        [ "_id" => '123', "value" => '2323' ]
];

But even if it were accepting my requests, it obviously wouldn't fetch any data because my e-mail address and password are nowhere to be found in that input data. I see no mention of e-mail addresses or passwords anywhere in your code either. What's the meaning of this? Don't tell me I have to use some massive library to support "SSE" in PHP... I just wanted to make a normal request with normal data. No need for it to be using this "SSE" stuff... but maybe that's the only way?

[charlag]

SSE is a very dumb protocol: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#Examples

If you check the source code, you will see that the push identifier must be created first with normal authentication: https://github.com/tutao/tutanota/blob/452106cfe7c8b3a4947bed70e2ed8a38e37a04d8/src/native/PushServiceApp.js#L84

If you don't want to jump through the hoops of creating and encrypting one by yourself, you can use existing client to do this for you and then reuse the identifier. Identifier together with userId is used to authenticate you. SSE connection is meant to be kept open so that you can get updates. I'm not sure how to do this in PHP.

It is obviously more involved than you want - sending email and password each time - but is also more powerful. We use sessions and principle of least privilege (leaking push identifier shouldn't leak mailbox data for example) so it might seem like over-engineered solution if you want to write one-liner in PHP Note that you need to send confirmation about received notifications back to the server.

[EAFSC]

All I can find is a "_id" value, but no "value" or "identifier" values. Have spent forever trying to get this to work now... even with just trying to reuse the website's info, which it hides extremely well...

[EAFSC]

Still stuck on this. I really can't figure it out based on the information provided, and I've stared myself blind on the linked code...

[charlag]

@EAFSC well, can you please be more specific in your questions? You need to:

1. Get your user id
2. Create push identifier on the server (you can generate any string as identifier itself) a. Using existing desktop or android app b. Manually, like here: https://github.com/tutao/tutanota/blob/master/src/native/PushServiceApp.js#L84 To do that you need to set up a session. LoginFacade does this.
3. Connect to SSE with your user id and that identifier string

Please describe what exactly you're doing and what is not working for you.

[EAFSC]

@charlag

1. "Get your user id"... Well, I think I was able to get the "user id" by looking around a lot in the network tab while logging in to Tutanota.
2. "Create push identifier on the server (you can generate any string as identifier itself)" <-- I don't understand this.

https://github.com/tutao/tutanota/blob/master/src/native/PushServiceApp.js#L84 <-- I don't understand this code (I don't mean syntax-wise, because I do know JavaScript). It makes all sorts of function calls which seems to suggest that I need to recode the entire Tutanota source code in PHP? I don't see anything I can actually translate to PHP there which doesn't cascade into endless complexity?

3. "To do that you need to set up a session. LoginFacade does this." <-- ?
4. "Connect to SSE with your user id and that identifier string" <-- Well, this last part seems like it would be the least problematic, but I need the JSON blob for it to work.

I don't understand why Tutanota wouldn't just have a simple JSON blob served without this "SSE" madness. Is it really meaningful to have people manually log in to their Tutanota accounts all the time, wasting your data traffic and system resources, only to get annoyed by the fact that there are no new e-mails? Seems like you'd actually have an interest in making it easy to see if there are any new e-mails so that logging in might be worthwhile. (It could just be a promotional e-mail from Tutanota that's new.)

I see this all the time these days. Even this site, GitHub, made for programmers, doesn't even provide any RSS/ATOM feeds for new replies. I had to code my own "weird hack" to be able to even track new replies in this issue, for example! It's madness to me. Inexplicable. I feel like nobody wants me to have any kind of control whatsoever, and I'm expected to manually check everything like a caveman.

[charlag]

@EAFSC I don't understand what you want us to do. Which BLOB do you want? SSE is already super dumb thing. What is complex is to

What most people do to get notified about new emails:

1. Use their mobile phone to log into Tutanota. They receive push notifications using SSE (Android) or APNS (iOS)
2. Use desktop client to receive notifications using SSE

What you do:

1. Go and complain that there's no magical way to generate some feed despite the fact that there is an SSE API and API to get emails. Session based authentication seems like a disadvantage to you.

So I don't understand your claim that everyone just logs in to check emails.

Okay, let's try to break it down. I'm not sure why do I have to do your homework for you but we're deep in it anyway.

How does it work on high level:

1. You create a PushIdentifier instance on the server. This is done so that server can identify and authenticate you in the future even without full-fledges session login.
2. You can SSE and you are notified.

Clearly the first part is what is hard right now. Let's break down login process. We start with creating a session: https://github.com/tutao/tutanota/blob/master/src/api/worker/facades/LoginFacade.js#L132 First of all we need to load passphrase key: https://github.com/tutao/tutanota/blob/master/src/api/worker/facades/LoginFacade.js#L370 To do that you need to make a request like ' GET https://mail.tutanota.com/rest/sys/saltservice?_body={"_format":"0","mailAddress":"you.email@tuta.io"}

So far so good?

Then we need to generate key from the password and salt. To do that, hash SHA256 your password. Grab salt field from the return of the previous request. Do BCrypt with password hash as password, salt as salt and 8 (eight) iterations. Result is your passphrase key. https://github.com/tutao/tutanota/blob/master/src/api/worker/crypto/Bcrypt.js#L32

Still with me? The hardest part is behind us fortunately.

Now we can prepare things to make an actual request to create a session. Create "auth verifier": hash with SHA256 your passphrase key and convert to Base64: https://github.com/tutao/tutanota/blob/master/src/api/worker/crypto/CryptoUtils.js#L47 After that make a POST to https://mail.tutanota.com/rest/sys/sessionservice with body JSON body like:

{
  "_format":"0",
  "accessKey":null,
  "authToken":null,
  "authVerifier":"AuthVerifierWhichYouMadeAtPreviousStep",
  "clientIdentifier":"My PHP script",
  "mailAddress":"your.email@tuta.io",
  "recoverCodeVerifier":null,
  "user":null
}

Hooray! Now we are really close. In response you get JSON like {"_format":"0","accessToken":"someTokenn","challenges":[],"user":"Jjxhv6U----0"} where accessToken is what you should hold into and user is your user id. You have a session now. Use it wisely.

Okay, creating a PushIdentifier instance. https://github.com/tutao/tutanota/blob/master/src/native/PushServiceApp.js#L84 First of all you need to load the User instance to get an ID of the list of push identifiers. Do GET on https://mail.tutanota.com/rest/sys/user/YourUserId with accessToken header being, unsurprisingly, accessToken. You get JSON with some stuff back and but you're looking for the pushIdentifierList field and inside of it grab list field.

You also need to get UserGroupInfo. You need to GET to address like https://mail.tutanota.com/rest/sys/groupinfo/firstIdPart/secondIdPart where parts of the ID are no the user.userGroup.groupInfo.

Then create a JSON like here: https://github.com/tutao/tutanota/blob/master/src/native/PushServiceApp.js#L84 display name can be whatever you want, pushServiceType is "3" for SSE, identifier is a random string generated by you. _owner and _ownerGroup is group field from UserGroupInfo we've grabbed at the first step. You do POST on https://mail.tutanota.com/rest/sys/pushidentifier/pushIdentifierListId with you JSON.

You can create email notification in Setting->Email->Notifications to spy on the request. You can also in theory skip all previous steps and just grab auth token and list ID from this request with different type and identifier.

Done! Now you can use SSE with that random string you generated. I was just following the flow of the functions I linked and verified some things in the browser. Hope this helps.

[EAFSC]

@charlag

It took many hours of hard work, but I was finally able to follow your instructions up until the "Hooray! Now we are really close." part, after which it gets really hairy and fuzzy... But I tried to run it so far to see that at least it returns the JSON data, only to get this error:

"password_hash(): Use of the 'salt' option to password_hash is deprecated"

Reading the manual and online, it turns out that PHP has deprecated or even already removed support for custom salts with BCrypt... claiming that "you could never make a more secure salt anyway". But obviously, this breaks my code because the custom salt is needed for Tutanota! This is not the thing I expected to get stuck on... and I don't know what to do now.

If this is some legacy thing, why does Tutanota use it? You seem to know what you're doing (especially since you apparently think that this stuff is easy!), so now I don't know what to do... Maybe you will soon change it to use some other algorithm or structure because of this? Should I really hunt down some bad library that will let me use the custom salt?

[charlag]

@EAFSC okay, listen. I don't know PHP. If you don'T know it though, I cannot help you much. I have no idea why they deprecated it, probably because people cannot provide proper salt. At the same time

password_hash() uses a strong hash, generates a strong salt, and applies proper rounds automatically. password_hash() is a simple crypt() wrapper and compatible with existing password hashes.

So you could theoretically do it by hand. Or use some library. I don't know. I did my part. But I recommend you to skip to the second to last parameter, spy on the request and just replace the parts you need.

[EAFSC]

At least you never got rude...

[EAFSC]

After wasting countless hours on this, 7 today alone, I'm now giving up. Tutanota clearly doesn't want to give this ability and have gone to great lengths to make it as cryptic and confusing as possible to figure out. Even with your descriptions, I can't get Tutanota's server to return anything for my requests to https://mail.tutanota.com/rest/sys/sessionservice . I've wasted too much time and energy on this now.

[charlag]

Okay, that's not nice. I spent quite some time helping you and I didn't hear a single "thank you". I literally sat down on my weekend to go through the steps with explanations and links to the code. I know that there are projects where people figured it out on their own. I even proposed you how to make a shortcut without creating a session on your own. I just want you to underhand three things:

1. "I don't understand this" doesn't mean "it's bad". It means there are reasons for it to be like that.
2. I don't have to help you. I'm just trying to be nice. I could ignore all of that.
3. We are not going to replace complex but reasonable e2e auth system to cater to your needs. And no one else would.