Open sealabr opened 7 years ago
You sent the link from my diff. That's working? Or was this a typo?
should we try this on the master?
On 17 August 2017 at 22:10, dietzi notifications@github.com wrote:
You sent the link from my diff. That's working? Or was this a typo?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tgalal/yowsup/issues/2149#issuecomment-323248114, or mute the thread https://github.com/notifications/unsubscribe-auth/AR0Z7uYzMX1Etj4UcPW6sE0qrbCZXhyuks5sZQC5gaJpZM4OWLf4 .
sorry i miss the link, this the link: https://pastebin.com/jxk0hrgU
@dietzi , have you found what's wrong ? it's seem that that whatsapp dest, cannot decrypt the image or i missed something?
I had no time to test your code. Maybe today I'll find time to test.
Im not sure if this help. I try dump the thumb_image data and deserialize it.
Another image field value:- media_url: https://mmg.whatsapp.net/d/f/AtYmOccQV4pO5TfQyeTDN0KZqVb9O6FfI4jL3VR7OJsP.enc media_enc_hash: Tgzo7Ycn/5mITjxvNHG2eBPeZ4M1orKP/9iJPXh/Uw0=
I think the problem is the encryption. If I click download it starts with downloading and after that the error appears. So the hash or the mediakey transmitted are wrong. We should test with some combinations of hash and encryption-key. Maybe Whatsapp expects the end-to-end-keys for encryption. Today in the evening I'll make some tests
@dietzi , i agree it something wrong when whatsapp decrypt the enc image. but strangely, yowsup can decrypt the image properly the same as if we sent from whatsapp. btw, i checked that there is 1 variables that exist in imageMessage protobuf when whatsapp that did'nt handle in current imageMessage struct, it's seems like hash file, but i don't know what is for/from.
for hash of the image it self, i ovserved it' should be valid , because it's give same value when i received.
and one more i found another hash in in protobuf imageMessage, may be it should filled with the hash of the encrypted image that we uploaded?
in wa.proto (see: hash_like)
message ImageMessage { required bytes url = 1; required string mime_type = 2; required string caption = 3; required bytes file_sha256 = 4; required uint64 file_length = 5; required uint32 height = 6; required uint32 width = 7; required bytes media_key = 8; optional bytes hash_like = 9; required bytes jpeg_thumbnail = 16; }
I've analysed your changes and I think the problem is at: message_media_downloadable.py: DownloadableMediaMessageProtocolEntity.fromBuilder(builder) ---> filehash = WATools.getFileHashForUpload2(builder.getFilepath()) I think the routine "getFileHashForUpload2" returns a wrong value. But how to figure it out?
hmm, but i got the same value with getFileHashForUpload2 hashfile if i send image from whatsapp and from yowsup, so i'm sure that getFileHashForUpload2 is right.
If I send a image I get this "error" in yowsup: Axolotl layer does not have the message, bubbling it upwards
So I think there is a Problem with the structre of the parameters or something else. I'm trying on. What I found in message_media_downloadable.py: the filepath for filesize doesn't include .enc - maybe this is the problem.
If I add "mediaNode.setAttribute("caption", "test")" in media_message_downloadable_image toProtocolTreeNode this works. Whatsapp download's the image but can not show it. So the filehash or the encryption may be wrong....
Hi guys, maybe we need to send both, the encrypted file-hash AND the unencrypted file-hash. I think it could be something like that:
message ImageMessage {
required bytes url = 1;
required string mime_type = 2;
required string caption = 3;
required bytes file_sha256 = 4;
required uint64 file_length = 5;
required uint32 height = 6;
required uint32 width = 7;
required bytes media_key = 8;
required bytes file_enc_sha256 = 9;
required bytes jpeg_thumbnail = 16;
}
Unfortunately, I don't know how to implement it, but maybe it helps...
@Nicos-git i've already done that also, but no diffrent in result. My suspicions now on : The refs for from and to number is wrong. because i observed that the refs always result the same hash for the same image. so it it function of : number and imagesize or imagehash
@bahtiarp
Could you please post your changes in a proper formating?
@ventrixcode https://pastebin.com/pVxryQ12
this is cmd to compile protobuf to python: protoc -I=. --python_out=. wa.proto
I'm testing your changes @ventrixcode and they look fine (at least they compile). I had couple of errors, some of them I could workaround, but not the last one.
A workaround: I'm in P3 and unicode() does not exist. So in wa_pb2.py I've added this at the beginning, below imports:
` if sys.version_info >= (3,):
def unicode(value,encoding):
if isinstance(value, str):
return value
else:
return value.decode(encoding)
`
Now, here is the fatal error while using /image send as usual. It fails because another process is being using a file (I suppose is the temporary image file). Here is the short error trace:
connected:/image send --number-is-hidden-- media\test.jpg DEBUG:yowsup.layers.logger.layer:tx:
b'33650d846c7ff15a4703ff85eb7102bc3bebd9172448e1bf76188d16f159ad4e' filesize:21258 21258 file_enc_sha256:ae9456c753fbd8967f2cfb79c36bac38700ff80713436cfa6f3ca30d70b9ef5d
{'size': '21258', 'filehash': 'rpRWx1P72JZ/LPt5w2usOHAP+AcTQ2z6bzyjDXC5710=', 'u rl': 'https://mmg.whatsapp.net/d/f/Ap_lt4XwJVOOFPI4wf67IefIthSCJ9-Z2cjICEI-8m9h. enc', 'name': None, 'type': 'encrypted', 'error': None, 'width': None, 'height': None, 'mimetype': 'application/octet-stream'}
ERROR:yowsup.common.optionalmodules:PIL import failed ERROR:yowsup.layers.protocol_media.mediauploader:Error occured at transfer [WinError 32] The process has no access to file because it is being used by another process: 'c:\users\--user-is-hidden--\appdata\local\temp\tmphxdp2p'
Here is the ful trace:
Traceback (most recent call last):
File "K:\aaa_git\home\xxx\git\yowsup\yowsup\yowsup\layers\protocol_media\mediauploader.py", line 221, in run
self.successCallback(sourcePath, self.jid, result)
File "K:\aaa_git\home\xxx\git\yowsup\yowsup\yowsup\demos\cli\layer.py", line 561, in
ERROR:yowsup.demos.cli.layer:Upload file media\test01.jpg to https://mmg.wh atsapp.net/u/f/3LTzkEotHb4SuIvIkeCc1FmkMngABVfRsyHuzQ/AiHJb7-OMAmEiCgrPe2Kv-X2oH MwDNaziol64AWvWLBy for --hidden--@s.whatsapp.net failed!
Any idea of what's going on? Are we realy doing something in parallel on such file, or maybe we forgot to close it?
Thanks. Regards. hgc.
I've found some wrong things according to Whatsapp Web code (I haven't tested it yet, will do it tomorrow).
*** I think there's something to do about cryptoKey**, like receiving it from the iqRequest response. But as I didn't test it, I don't know yet.
def getMediaLabel(self):
if self.media_type == "image":
m_type = "Image"
elif self.media_type == "video":
m_type = "Video"
elif self.media_type == "audio":
m_type = "Audio"
elif self.media_type == "document":
m_type = "Document"
return "WhatsApp %s Keys" % m_type
def generateCrypto(self):
cryptoKey = binascii.hexlify(self.b64Hash) # don't know if it's right, maybe another key
derivative = HKDFv3().deriveSecrets(binascii.unhexlify(cryptoKey), self.getMediaLabel(), 112)
iv = derivative[:16]
cipherKey = derivative[16: 16 + 32] # length is 32
macKey = derivative[48: 48 + 32]
refKey = derivative[80: 80 + 32]
# instance variables (returned on success)
self.refKey = refKey
# return a function to encode image
return AES.new(key=cipherKey, mode=AES.MODE_CBC, IV=iv)
When you create phone refs, you need to encrypt phone with "@s.whatsapp.net" using HmacSha256 and the refKey.
def encryptPhone(self, jid):
# generate hashs for header request "refs"
sha1 = hashlib.sha256(self.refKey)
parsed_jid = Jid.normalize(jid).replace("@c.us", "@s.whatsapp.net").encode("utf-8")
sha1.update(parsed_jid)
return base64.b64encode(sha1.digest()[:20])
I'll be waiting your tests. Thanks!
i think cryptoKey should be: "576861747341707020496d616765204b657973" based on decrypt method.
@bahtiarp no, this key you've sent is just an encoded string, like in getMediaLabel. Look:
I still do not have a phone number, I lost another one while registering (don't know why); I will try again later.
I've send a image and could get this from whatsapp-web:
Assertion Failed! decryption-error (false not truthy): msg:processEncryptedMedia MediaEncryptionError: hmac fail computed: DI97Dct8dKIyPA== given: jM9RL+ZJtDYMiQ==
Can we compute the right key back?
This is the function from whatsapp-javascript:
function d(e, t, n, a, o, i) { return f["default"].resolve().then(function() { if (!(10 <= o.length && o.length <= 32)) throw new v["default"].MediaEncryptionError("Given bad expected hmac"); var i = m["default"].build(t, n).readByteArray(); return new r(a).sign(i).then(function(t) { for (var n = new Uint8Array(t, 0, o.length), a = !1, r = 0; r < n.length; r++) n[r] !== o[r] && (a = !0); if (a) throw new v["default"].MediaEncryptionError("hmac fail computed: " + h["default"].encode(n) + " given: " + h["default"].encode(o)); return c(e, i) }) }).then(function(e) { var t = "string" == typeof i ? i : i(new Uint8Array(e)); return w["default"].createFromData(e, t) }) }
If I change the refTo and refFrom the computed value in WhatsApp changes. So we should search the problem there
Update: the keys change on every send.
Some tests later I can't solve the problem. I can't find the given hmac in yowsup. Where should this key be in yowsup? The WhatsApp-log say "MediaEncryptionError". So the function "encryptImg" seem's to be false. For now it's enough today. I'll go to bed now. Good night at all.
@dietzi , interesting progress. can you tell me how to get that js log in chrome. it'll will help for dubuging. so far i didn't see error log appeared in console when inspected in chrome.
I'm using firefox. There I press ctrl+shift+k and then on analyse network. WhatsApp tries to load the image and on error I see a POST-request with the log as response. Sometimes I have to refresh the page. I'll post here a image
@dietzi , thanks. i'll try to do some experiment tonight.
I think we missing hmacKey encryption in yowsup. Please refer my comment above.
Found article about hmackey
https://codedump.io/share/hGCbxQ8vsuoT/1/decrypting-aes-and-hmac-with-pycrypto
Great example and code on decrypting! Thanks! But I don't know if I'll be able to apply it. Let me see.
Not sure how useful this is but the whatsapp-plugin for the multi-messenger Disa can send Media without problems.
It's closed source but fortunately written in c# which can be decompiled basically perfectly (if anyone wants to do further digging the weird dpk can be unpacked using bindump -e file.dpk
binwalk -e file.dpk
and the dlls can be decompiled using dnSpy (or probably any other c# decompiler).
This is the decompiled UploadFile method from the plugin:
// Disa.Framework.WhatsApp.WhatsApp
// Token: 0x06000518 RID: 1304 RVA: 0x0001AAC4 File Offset: 0x00018CC4
private ProtocolTreeNode UploadFile(string to, string fileType, string fileHash, byte[] fileBytes, string filePath, byte[] thumbnail, string mimeType, Action<int> progressUpdate, bool recording = false, int seconds = 0, bool encrypted = false, Dictionary<string, string> extraUploadStrings = null)
{
Func<string> func = delegate
{
string mimeType2 = mimeType;
uint num = <PrivateImplementationDetails>.ComputeStringHash(mimeType2);
if (num <= 3109025507u)
{
if (num <= 2471643833u)
{
if (num != 1639502448u)
{
if (num != 1826612852u)
{
if (num == 2471643833u)
{
if (mimeType2 == "audio/mp4")
{
return ".mp4";
}
}
}
else if (mimeType2 == "audio/wav")
{
return ".wav";
}
}
else if (mimeType2 == "video/mp4")
{
return ".mp4";
}
}
else if (num != 2799670939u)
{
if (num != 2970156561u)
{
if (num == 3109025507u)
{
if (mimeType2 == "audio/mpeg")
{
return ".mp3";
}
}
}
else if (mimeType2 == "audio/aac")
{
return ".aac";
}
}
else if (mimeType2 == "audio/ogg; codecs=opus")
{
return ".opus";
}
}
else if (num <= 3404529107u)
{
if (num != 3355350250u)
{
if (num != 3373101051u)
{
if (num == 3404529107u)
{
if (mimeType2 == "video/3gpp")
{
return ".3gp";
}
}
}
else if (mimeType2 == "audio/webm")
{
return ".webm";
}
}
else if (mimeType2 == "audio/amr")
{
return ".amr";
}
}
else if (num != 3485255564u)
{
if (num != 3901389917u)
{
if (num == 4015067401u)
{
if (mimeType2 == "video/avi")
{
return ".avi";
}
}
}
else if (mimeType2 == "image/jpeg")
{
return ".jpg";
}
}
else if (mimeType2 == "audio/3gpp")
{
return ".3gp";
}
throw new Exception("Unsupported MimeType to determine file extension");
};
List<KeyValue> list = Enumerable.ToList<KeyValue>(new KeyValue[]
{
new KeyValue("hash", fileHash),
new KeyValue("type", fileType)
});
if (!encrypted)
{
list.Add(new KeyValue("size", ((filePath == null) ? ((long)fileBytes.Length) : new FileInfo(filePath).Length).ToString(CultureInfo.InvariantCulture)));
}
ProtocolTreeNode children = new ProtocolTreeNode((!encrypted) ? "media" : "encr_media", list.ToArray());
string text = this.MakeIqId("request_media_upload_");
ProtocolTreeNode node = new ProtocolTreeNode("iq", new KeyValue[]
{
new KeyValue("id", text),
new KeyValue("type", "set"),
new KeyValue("to", "s.whatsapp.net"),
new KeyValue("xmlns", "w:m")
}, children);
WhatsApp.WaitTask3<bool, ProtocolTreeNode> waitTask = new WhatsApp.WaitTask3<bool, ProtocolTreeNode>(30000, text);
this._uploadResponses.Add(waitTask);
try
{
this.SendProtocolTreeNode(node, false);
}
catch
{
base.DebugPrint("Failed to ask for an upload.");
this._uploadResponses.Remove(waitTask);
throw;
}
if (!waitTask.WaitOne())
{
this._uploadResponses.Remove(waitTask);
throw new TimeoutException("Eish. Waited too long for a upload request.");
}
ProtocolTreeNode additional = waitTask.Additional;
Utils.DebugPrint(additional.NodeString(""));
if (additional.GetChild("duplicate") != null)
{
ProtocolTreeNode child = additional.GetChild("duplicate");
string attribute = child.GetAttribute("url");
string attribute2 = child.GetAttribute("size");
return WhatsApp.GenerateMediaSendNode(child.GetAttribute("type"), attribute, Enumerable.Last<string>(attribute.Split(new char[]
{
'/'
})), attribute2, thumbnail, recording, seconds);
}
if (additional.GetChild((!encrypted) ? "media" : "encr_media") == null || string.IsNullOrWhiteSpace(additional.GetChild((!encrypted) ? "media" : "encr_media").GetAttribute("url")))
{
throw new ServiceBubbleSendFailedException("Could not get a url from the node.");
}
string attribute3 = additional.GetChild((!encrypted) ? "media" : "encr_media").GetAttribute("url");
string name = this.GenerateMediaFilename(func.Invoke());
Dictionary<string, object> dictionary = new Dictionary<string, object>();
dictionary.Add("to", to);
dictionary.Add("from", this.GetMyAccountWaAddress());
Dictionary<string, object> dictionary2 = dictionary;
if (filePath == null)
{
dictionary2.Add("file", new OkHttpClient.FormFile
{
Name = name,
ContentType = mimeType,
Bytes = fileBytes
});
}
else
{
dictionary2.Add("file", new OkHttpClient.FormFile
{
Name = name,
ContentType = mimeType,
FilePath = filePath
});
}
if (extraUploadStrings != null)
{
foreach (KeyValuePair<string, string> keyValuePair in extraUploadStrings)
{
dictionary2.Add(keyValuePair.Key, keyValuePair.Value);
}
}
using (OkHttpClient okHttpClient = new OkHttpClient())
{
if (mimeType.StartsWith("image"))
{
okHttpClient.Timeout = 30000;
okHttpClient.ReadWriteTimeout = 30000;
}
else
{
okHttpClient.Timeout = 60000;
okHttpClient.ReadWriteTimeout = 60000;
}
okHttpClient.UserAgent = WhatsConstants.UserAgent;
try
{
string text2 = okHttpClient.PostMultiPart(attribute3, dictionary2, delegate(double progress)
{
progressUpdate.Invoke((int)progress);
});
Utils.DebugPrint(text2);
string[] array = text2.Split(new string[]
{
"\n"
}, 1);
int i = 0;
while (i < array.Length)
{
string text3 = array[i];
if (text3.StartsWith("{"))
{
JObject jobject = JObject.Parse(text3.TrimEnd(new char[1]));
if (string.IsNullOrWhiteSpace((string)jobject["filehash"]))
{
throw new Exception("File hash is null!");
}
string text4 = (string)jobject["name"];
string text5 = (string)jobject["url"];
if (string.IsNullOrWhiteSpace(text4))
{
text4 = Enumerable.Last<string>(text5.Split(new char[]
{
'/'
}));
Utils.DebugPrint("No filename. Generated: " + text4);
}
return WhatsApp.GenerateMediaSendNode((string)jobject["type"], text5, text4, (string)jobject["size"], thumbnail, recording, seconds);
}
else
{
i++;
}
}
}
catch (OkHttpClient.UploadMultiPartException ex)
{
if (ex.Status == 415)
{
throw new ServiceBubbleSendFailedException("Unsupported media type.");
}
throw;
}
}
throw new ServiceBubbleSendFailedException("Could not upload file (end of method).");
}
@jakibaki , can you find / decompile function: GenerateMediaSendNode. (And functions related to encrypt an image/media) and how to install bindump.
@jakibaki , can you find / decompile function: GenerateMediaSendNode.
Here's a zip with all the code decompiled and the original dlls.
and how to install bindump.
Woops I meant binwalk, it's in the package repos of most linux-distros and for macOS in homebrew. There is no windows-support afaik.
So this is the GenerateMediaSendNode Method:
// Token: 0x06000514 RID: 1300 RVA: 0x0001A910 File Offset: 0x00018B10
private static ProtocolTreeNode GenerateMediaSendNode(string type, string url, string name, string size, byte[] thumbnail, bool recording = false, int seconds = 0)
{
List<KeyValue> list = new List<KeyValue>();
list.Add(new KeyValue("type", type));
list.Add(new KeyValue("url", url));
list.Add(new KeyValue("file", name));
list.Add(new KeyValue("size", size));
List<KeyValue> list2 = list;
if (thumbnail != null)
{
list2.Add(new KeyValue("encoding", "raw"));
}
if (recording)
{
list2.Add(new KeyValue("origin", "live"));
}
if (seconds > 0)
{
list2.Add(new KeyValue("seconds", seconds.ToString(CultureInfo.InvariantCulture)));
}
return new ProtocolTreeNode("media", list2, null, thumbnail);
}
@jakibaki, thanks. finally i've to installed vs2017 and learn c#, only for patching 3 lines of my last code. but it's working now to send an image via yowsup. i'll show the diff soon.
Thank you so much!
Please Tell us how to apply your changes!
I still didn't have time to tidy up the code, but this is the change i've made:
def pad(self,s):
return s + (16 - len(s) % 16) * chr(16 - len(s) % 16)
def encryptImg(self,img, refkey):
derivative = HKDFv3().deriveSecrets(binascii.unhexlify(refkey),
binascii.unhexlify("576861747341707020496d616765204b657973"), 112)
parts = ByteUtil.split(derivative, 16, 32)
iv = parts[0]
cipherKey = parts[1]
macKey=derivative[48:80]
mac = hmac.new(macKey,digestmod=hashlib.sha256)
mac.update(iv)
cipher = AES.new(key=cipherKey, mode=AES.MODE_CBC, IV=iv)
imgEnc = cipher.encrypt(self.pad(img))
mac.update(imgEnc)
hash = mac.digest()
hashKey = ByteUtil.trim(mac.digest(), 10)
finalEnc = imgEnc + hashKey
return finalEnc
@bahtiarp What is this string "576861747341707020496d616765204b657973" you are using in line 6?
And how can I send the encrypted file (finalEnc)?
like in c#
string text = null;
if (bubble is ImageBubble)
{
text = "WhatsApp Image Keys";
}
if (bubble is VideoBubble)
{
text = "WhatsApp Video Keys";
}
if (bubble is AudioBubble)
{
text = "WhatsApp Audio Keys";
}
if (bubble is FileBubble)
{
text = "WhatsApp Document Keys";
}
@bahtiarp Okay thanks but still don't get how I can now send an image. I created now my encrypted image but which method in yowsup I have to use to send the image to a Jid?
Great. It work's. Now we have to copy this for video, audio and documents. I need to send audio. Maybe tomorrow I will try to copy this lines for audio.
@bahtiarp Is that from above the final diff or is it following? I ask because some people said it works.
Im too dumb to get this working if anyone could post all the changes in all files?
They currently seem to tidy up the code so we don't need to implement unnecessary stuff. This was a long term work thread :)
Please make an new pull request later then or better make an own fork with patched stuff @bahtiarp
Thank you ;)
Hello I'm trying to upload media by yowsup, using
RequestUploadIqProtocolEntity
but it's not possible because lambda function allways direct toonRequestUploadError
, and cant send file ...Request upload for file /tmp/jpeg.jpg for 5524XXXXXXXX@s.whatsapp.net failed
I also tried with yowsupCLI using image send
/image send 5524XXXXXX /tmp/3DCDA9A9436B01F9A3.jfif