Send media - Request upload for file XXXX@s.whatsapp.net failed [BOUNTY FOR SOLUTION]

[sealabr]

Hello I'm trying to upload media by yowsup, using RequestUploadIqProtocolEntity but it's not possible because lambda function allways direct to onRequestUploadError, and cant send file ...

Request upload for file /tmp/jpeg.jpg for 5524XXXXXXXX@s.whatsapp.net failed

I also tried with yowsupCLI using image send

/image send 5524XXXXXX /tmp/3DCDA9A9436B01F9A3.jfif

ERROR:yowsup.demos.cli.layer:Request upload for file /tmp/3DCDA9A9436B01F9A3.jfif for 5524XXXXXXX@s.whatsapp.net failed
DEBUG:yowsup.layers.logger.layer:rx:
<iq type="error" from="s.whatsapp.net" id="1">
<error text="feature-not-implemented" code="501">
</error>

[pabloruan0710]

I too have same problem.

[Bartemis]

I have the same problem, looks like a problem where whatsapp changed their method..

[etinin]

Just solved part of the issue:

The tag is no longer supported. Now you must send encrypted images with .

If you make the needed changes in iq_requestupload.py and iq_requestupload_result.py (replace "media" with "encr_media", you can get to the upload step.

I could not, however, get the upload to actually work. Server always gives 400 error.

Maybe the server checks if the image is actually encrypted?

Gotta check some of the forks which can actually decrypt the image to see if we can figure out a procedure for encryption.

I'm trying to get the correct POST format from the decompiled java source, with no success.

If anyone has any idea how to proceed, I'd be immensely grateful.

[sealabr]

Glad to hear that someone is trying to do something, now I know that is needed to send encryped media, instead of plain text media, as lguardi commented on

2123

Basically, you need to implement the inverse of the send procedure: 
- Generate a random key for encryption 
- Encrypt media and upload to the server 
- send encrypted url to recipient and the encryption key in a protobuf message

I'm looking forward to understand more about encrypted protocol in yowsup, to actually do something about send encrypted media protocol, I will be glad to any help given.

[bahtiarp]

@etinin , i follow you to change "media" to "encr_media", got the upload URL, but then got this post & response:

POST: https://mmg.whatsapp.net/u/f/FIcI1bQcJGbRXaGee0_cSllrJNoABVRrIxaEqA/AkObQlFqUS5DA21KPJHyYOnL47WKzVMSGN8Jf0SaiBE8 Content-Type: multipart/form-data; boundary=zzXXzzYYzzXXzzQQ Host: mmg.whatsapp.net User-Agent: WhatsApp/2.16.11 S40Version/14.26 Device/Nokia-302 Content-Length: 4880

HEADER & FOOTER: --zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="from"

6281315318608@s.whatsapp.net --zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="file"; filename="d6d02600f4f65c6733303c90a821800e.jpg" Content-Type: image/jpeg

RESPONSE: HTTP/1.1 400 Bad Request Content-Type: application/json Date: Sat, 15 Jul 2017 17:10:05 GMT Connection: close Content-Length: 0

It seems enc image is send in the protobuf message, like when extracted image from receiving image. I think @jlguardi , can help give us some more clues.

[etinin]

I'm currently short of the required time, but if someone could take a look at jiguardi's fork, this might be solved. As I understand it, his fork has the decryption method properly implemented.

https://github.com/jlguardi/yowsup/tree/master/yowsup

In a parallel note, maybe we could try uploading a pre-encrypted image file? As I understand, the server should probably accept it, since it doesn't really have the key. But, again, I haven't studied the proper code.

It seems you just have to send yourself an encrypted image and check the specifics of the AES encryption being employed.

I've tried intercepting the whatsapp app requests to solve this issue but it's harder than I believed. The APP both on iPhone and Android seems to ignore proxy settings entirely when it can connect straight to the internet. And even some requests that get through the proxy are closed by Whatsapp when the app sees the selfsigned SSL certificate (even though the cert is properly added to the system.

Anyone has knowledge any reliable procedure for intercepting requests? Med school is keeping me way too busy to keep working blindly.

[bahtiarp]

I made some modifications, to send back received image with enc, But recipient only got this message in WA instead of the image: Waiting for this message. This may take a while. Learn More

This my debug log:

[message to="62851008xxxx@s.whatsapp.net" type="media" id="1500356450-2"] [enc type="msg" mediatype="image" v="2"] 3! �ƕ�u����[ -�J�,��q���z�/�6wE"�� �MQ�H�x�Y�6��� .....BLABLA HEX:330a2105845ae2d89594e ...BLABLA [/enc] [/message]

2017-07-18 12:40:51,414 [MainThread ] [DEBUG] rx: [ack t="1500356451" from="628510080xxxx@s.whatsapp.net" class="message" id="1500356450-2"] [/ack]

2017-07-18 12:40:52,778 [MainThread ] [DEBUG] rx: [receipt type="retry" from="628510080xxx@s.whatsapp.net" id="1500356450-2" t="1500356452"] [retry count="1" t="1500356451" id="1500356450-2" v="1"] [/retry] [registration] ('+ê HEX:28272bea [/registration] [/receipt]

2017-07-18 12:40:52,779 [MainThread ] [DEBUG] tx: [ack to="628510080xxxx@s.whatsapp.net" type="retry" id="1500356450-2" class="receipt"] [/ack]

[etinin]

@bahtiarp, you've got to properly encrypt the image and send the encryption key. How did you get to this point? Are you uploading the image or just resending something you received?

Could you please send the patch?

[langioletto]

https://github.com/tgalal/yowsup/issues/2016#issuecomment-317543592

[bahtiarp]

@etinin

This is how to encrypt:

def encryptImg(img, refkey): img = pad(img) derivative = HKDFv3().deriveSecrets(binascii.unhexlify(refkey), binascii.unhexlify("576861747341707020496d616765204b657973"), 112) parts = ByteUtil.split(derivative, 16, 32) iv = parts[0] cipherKey = parts[1] cipher = AES.new(key=cipherKey, mode=AES.MODE_CBC, IV=iv) imgEnc=cipher.encrypt(img) return imgEnc[:-6]

refkey = binascii.hexlify(os.urandom(112))

[dietzi]

Where should this be? In mediauploader.py after "stream = f.read()" ? I got this error: global name pad is not defined. Should I also change the media-tag to encr_media ?

[pabloruan0710]

@dietzi try this

def pad(self, s):
        return s + ((16-len(s) % 16) * '{')

[dietzi]

Here is my diff: https://pastebin.com/mWvjhSUX Now I get "json data not found" at mediauploader.py line 160

[pabloruan0710]

try this:

after ssl_sock.write(bytearray(fBAOS.encode()))
dataux = ssl_sock.recv()

lines = dataux.decode().splitlines() 

[bahtiarp]

@dietzi

If you see the log, you got this when you upload the data:

RESPONSE: HTTP/1.1 400 Bad Request Content-Type: application/json Date: Sun, 16 Jul 2017 13:24:23 GMT Connection: close Content-Length: 0

(so you dont get the json data, as expected by the next processed).

I've checked from wa web version (chrome).

I see the header when uploading encr image is diffrent from yowsup mediauploader done.

yowsup mediauploader.py version for Header POST:

POST https://mmg.whatsapp.net/u/f/Gqp_7Z4ZN1PsLrg8b3xHGllraQYABVRvM1QKAg/AkObQlFqUS5DA21KPJHyYOnL47WKzVMSGN8Jf0SaiBE8 Content-Type: multipart/form-data; boundary=zzXXzzYYzzXXzzQQ Host: mmg.whatsapp.net User-Agent: WhatsApp/2.16.11 S40Version/14.26 Device/Nokia-302 Content-Length: 4892

--zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="to"

628510xxxxxx@s.whatsapp.net --zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="from"

628510xxxxxx@s.whatsapp.net --zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="file"; filename="AkObQlFqUS5DA21KPJHyYOnL47WKzVMSGN8Jf0SaiBE8.enc" Content-Type: image/jpeg

Response from Whatsapp: HTTP/1.1 400 Bad Request Content-Type: application/json Date: Sun, 16 Jul 2017 13:24:23 GMT Connection: close Content-Length: 0

Whatsapp WEB (chrome) version for Header POST:

------WebKitFormBoundary83mOQO9W0A1qhgX7 Content-Disposition: form-data; name="hash"

xxfuy7CpXoeRjSI321m74C8sXG/u3Bls8niwzzdGxuE= ------WebKitFormBoundary83mOQO9W0A1qhgX7 Content-Disposition: form-data; name="refs"

UL9-EipnOTCQuDjlaN-C5_pwW5k= p64bypqos9A0WxZrNLYkUOg2oZ4= ------WebKitFormBoundary83mOQO9W0A1qhgX7-- Content-Disposition: form-data; name="file"; filename="blob" Content-Type: application/octet-stream

Whatsapp WEB (chrome) RESPONSE: {"type":"encrypted","mimetype":"application/octet-stream","size":"7034","filehash":"xxfuy7CpXoeRjSI321m74C8sXG/u3Bls8niwzzdGxuE=","url":"https://mmg-fna.whatsapp.net/d/f/AsL21rfdOsW-bDtnaMAHOcSsmWgt2gjQAtCfwiELT_Mj.enc"}

i've checked that this: xxfuy7CpXoeRjSI321m74C8sXG/u3Bls8niwzzdGxuE= is the b64Hash of image file.

While this two (Content-Disposition: form-data; name="refs"): UL9-EipnOTCQuDjlaN-C5_pwW5k= p64bypqos9A0WxZrNLYkUOg2oZ4= I still dont know how to genereate.

But although we got the correct response from whatsapp when uploaded, we have totally have to change the next process in yowsup. because it's diffrent from old responses that yowsup processed.

[pabloruan0710]

@bahtiarp The refs is generated using the send number and recipient number with encripty:

• This is in JS, need rewrite in Python

https://cdnjs.cloudflare.com/ajax/libs/crypto-js/3.1.9-1/crypto-js.min.js
https://cdnjs.cloudflare.com/ajax/libs/crypto-js/3.1.9-1/sha256.min.js
https://cdnjs.cloudflare.com/ajax/libs/crypto-js/3.1.9-1/enc-utf8.min.js
https://cdnjs.cloudflare.com/ajax/libs/crypto-js/3.1.9-1/enc-base64.min.js
https://cdnjs.cloudflare.com/ajax/libs/crypto-js/3.1.9-1/lib-typedarrays.min.js

e = "55628226****@c.us" 
for (var t = e.length, n = [], r = 0; t > r; r++)
n[r >>> 2] |= (255 & e.charCodeAt(r)) << 24 - r % 4 * 8;

a = new Uint8Array(); 
b = CryptoJS.lib.WordArray.create(a)

var t = CryptoJS.HmacSHA256(CryptoJS.enc.Utf8.parse(e), b); 
t.sigBytes = 20
t.clamp()
var e = t.toString(CryptoJS.enc.Base64)
alert(e) 
console.log(e)

[dietzi]

If I add "hash" to the header with a random key then I get "unsupported media type". I think if we could get the hash-code then we will be able to send media. Does anybody know about to generate the hash?

[bahtiarp]

@pabloruan0710 using your js script i still cannot get the correct refs;

from: 62851xxxxxx to: 62895330xxxxx

refs should be:

UL9-EipnOTCQuDjlaN-C5_pwW5k= qohsee679xXFE8N_aXqB40PLX1U=

can you try to get the same result with your js script?

[pabloruan0710]

@bahtiarp This numbers ref is random Cripty, not is same number, check post with sniffer in Whatsapp web.

[pabloruan0710]

I can Send, and received this response:

Content-Type: multipart/form-data; boundary=zzXXzzYYzzXXzzQQ
Host: mmg.whatsapp.net
User-Agent: WhatsApp/2.16.12 S40Version/14.26 Device/Nokia-302
Content-Length: 41905
HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 08 Aug 2017 19:19:51 GMT
Connection: close
Content-Length: 221
{"type":"encrypted","mimetype":"application/octet-stream","size":"41517","filehash":"Ll9P92iz1ERZHgM1pSflrGT0UU6RUEzU4fTAE95JkA8=","url":"https://mmg-fna.whatsapp.net/d/f/AoEGRkSJG36LMUwEjvkGtUyjcNLUq8ftLcqu2Xf9EMZt.enc"}

However didn't received image in recipient number

[jlguardi]

@pabloruan0710 Have you sent ProtocolMedia message to the recipient? Or you have just uploaded the file.

[duzzifelipe]

@pabloruan0710 What did you do with refs? Did you encrypt sender and recipient respectively?

[bahtiarp]

@pabloruan0710 great you've succeed to upload the image and get the response. (i still figuring out how to get the "refs" correct. unless you show the code :) )

The next step is like @jlguardi said, you have todo some modification in protocol media (toProtocolTreeNode), so the message will send in the following protobuf format:

[message to="62851008xxxx@s.whatsapp.net" type="media" id="1500356450-2"] [enc type="msg" mediatype="image" v="2"] 3! �ƕ�u����[ -�J�,��q���z�/�6wE"�� �MQ�H�x�Y�6��� ..... --> DATA [/enc] [/message]

where DATA consist (encrypted) of: random mediaKey, url (we got when uploaded)

[whyasta]

sorry for my bad english.

@bahtiarp i can send the media message (without encrypt img) and response similar like @pabloruan0710 , the recipient never received the message. But if i send the encrypted image, always return 400 bad request. To generate the refs, i made some modifications of hBAOS :

    digTo = hmac.new("".encode("utf-8"), self.jid.replace("@s.whatsapp.net","@c.us").encode("utf-8"), hashlib.sha256).digest()[:20]
    refTo = base64.b64encode(digTo).decode()
    digFrom = hmac.new("".encode("utf-8"), self.accountJid.replace("@s.whatsapp.net","@c.us").encode("utf-8"), hashlib.sha256).digest()[:20]
    refFrom = base64.b64encode(digFrom).decode()

    hBAOS = "--" + boundary + "\r\n"
        hBAOS += "Content-Disposition: form-data; name=\"hash\"\r\n\r\n"
        hBAOS += self.__class__.getFileHashForUpload(sourcePath) + "\r\n"

        hBAOS += "--" + boundary + "\r\n"
        hBAOS += "Content-Disposition: form-data; name=\"refs\"\r\n\r\n"
        hBAOS += refFrom + "\r\n"
        hBAOS += refTo + "\r\n"

        hBAOS += "--" + boundary + "\r\n"
        hBAOS += "Content-Disposition: form-data; name=\"file\"; filename=\"" + "blob" + "\"\r\n"
        hBAOS += "Content-Type: " + "application/octet-stream" + "\r\n\r\n"

How to encrypt image properly? Any help would be appreciated.

[duzzifelipe]

@whyasta I think you're getting "400 bad request" because the length of encrypted file isn't the same from the raw file, so your "filesize" needs to be equal "len(stream)".

[bahtiarp]

@whyasta

It's seems that that your Content-Length is based on file before encrypted.

This is how i fixed it:

    try:
        filename = os.path.basename(sourcePath)
        filetype = MimeTools.getMIME(filename)
        f = open(sourcePath, 'rb')
        stream = f.read()
        f.close()
        refkey = binascii.hexlify(os.urandom(112))
        stream=self.encryptImg(stream,refkey)
        fenc = open(filename+".enc", 'wb')  
        fenc.write(stream)
        fenc.seek(0, 2)
        filesize=fenc.tell()
        fenc.close()

        sha1 = hashlib.sha256()
        sha1.update(stream)
        b64Hash = base64.b64encode(sha1.digest())

Now we can continue to fixing the next process.

[duzzifelipe]

@bahtiarp did you get to successfully upload the file? I'm stopped on an error saying "415 Unsupported Media Type"

[bahtiarp]

@duzzifelipe Yes, i got this:

POST https://mmg.whatsapp.net/u/f/ctq7Cm2nDwGv6rv1jqGWdlmMe7gABVZn2tCVcQ/AmJ99kyxGcMaIS1BkYC8vyFDFmPPbXQ779hJKmhWMCZL Content-Type: multipart/form-data; boundary=zzXXzzYYzzXXzzQQ Host: mmg.whatsapp.net User-Agent: WhatsApp/2.16.11 S40Version/14.26 Device/Nokia-302 Content-Length: 7422

--zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="hash"

NCzF2z0pL8F+zAN41s/MFiJPOrL4yi0cNMmCsWSHc6w= --zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="refs"

bsD9smTr55H5ifQkU3xd+F1Webo= KuU/zMxPnXiAqbatZ0tN3aKXjqA= --zzXXzzYYzzXXzzQQ Content-Disposition: form-data; name="file"; filename="blob" Content-Type: application/octet-stream

RESPONSE: HTTP/1.1 200 OK Content-Type: application/json Date: Thu, 10 Aug 2017 15:28:56 GMT Connection: close Content-Length: 220

{"type":"encrypted","mimetype":"application/octet-stream","size":"7034","filehash":"NCzF2z0pL8F+zAN41s/MFiJPOrL4yi0cNMmCsWSHc6w=","url":"https://mmg-fna.whatsapp.net/d/f/AtOdjxzg-CAUEi-2LL_uepNfYNquO9rIItlkp-uKRngh.enc"}

As you can see: Content-Length: 7422 --> size of the encrypted file + headers length "size":"7034" --> original size of the encrypted file return by whatsapp.

[duzzifelipe]

@bahtiarp woow! I got it using your code to generate b64Hash. Thank you! Now I'll try to see how this message is sent by whatsapp, maybe using this plugin: https://github.com/jlguardi/wireshark-whatsapp

[nfsprodriver]

Can you help us, the Loqui team, to get this working under our messenger LoquiIM? It's based on yowsup, but uses javascript as language. You can join our telegram group. Thank you, that we can see some work on yowsup again!

[dietzi]

Does anybody know what to do for sending the image to the receipent? The file is uploading correctly but not sent to the other number. There is no error or something else. What message must be send to push the image to the receipent? Here is my diff: https://pastebin.com/LhtH7GyX

[dietzi]

See my diff: https://pastebin.com/ADJX5EnQ Now I'm getting an ack-message from the receipent but there is no message. It seems the message is sent but it never comes to the phone... See my log: https://pastebin.com/ettx3t0p

[bahtiarp]

@dietzi

btw why is it twice to send same request for upload: (see line 3 and 9 on your log) and also you get 2 same rx for those request: line 15 and line 39.

[dietzi]

I don't know. The log says the message is send but it never retrieves on the phone..... I think the error is at line 32 it should be or something else....

[bahtiarp]

if you see my post 26 days ago, and follow the same format your message will arrive in dest phone. the encrypted message consist of :

1. protobuf ImageMessage structure -> mainly consist of: url and mediaKey.
2. image preview.

I'm still looking for my idle time to learn and modify in axolot layer for sending encrypted image message, or waiting for others solved this first.

[dietzi]

What did you changed in the code? Can you show a diff?

[bahtiarp]

i made some modifications but i forgat all, but mainly i modify https://github.com/tgalal/yowsup/blob/master/yowsup/layers/axolotl/layer_send.py line 46, so that the message will send in that format.

[dietzi]

Just go to console in your yowsup-folder and type "git diff"

[dietzi]

@bahtiarp

You wrote in your post (https://github.com/tgalal/yowsup/issues/2149#issuecomment-322039950) "encrypted message consist of...". Have the message to be encrypted? Yowsup sends this message after upload (I changed the code so mediaKey (got from refkey) is included):

ID: 1502735729-2 To: 49176****@s.whatsapp.net Type: media Timestamp: 1502735729 Media Type: image Has Preview: True MimeType: image File Hash: 3736a38324a39507852765042775a37514a76564c6a57432f564761626156576e7544582b5074695457593d URL: https://mmg.whatsapp.net/d/f/ApbBa3ek7yXDgkMaPr3gyYZYAcKNDIWTq3zvSRTmQ1D4.enc IP: None File Size: 5136 File name: i.png Encoding: raw Width: 192 Height: 192 mediaKey: 99c1863d363939796dcf66b6161cc0ed56d12bc2a282a0c132c8c5178d46b3865e34cec0f8aa92e8e5afc233f32df93bfd80a0b010e165f6f2796c5033eab2a7c12236709b12647d2733dee9f294f03c411c89a7db9f25ab45445883ccc23554dda99c970bbc2a3d55e9819cdfb6c3aa

`

But on my phone I only get the message "Waiting for this message......". Yowsup shows "\".

[bahtiarp]

Base on @jlguardi hint: We need the inverse procedure when receiving image:

• Generate a random key for encryption (done)
• Encrypt media and upload to the server (done)
• send encrypted url to recipient and the encryption key in a protobuf message (doable)

this is my log when receiving encrypted message with url to encrypted image: image-received-log this is my log when sent data: where [DATA] in body of mediaNode is encrypted of protobuf imageMesage.

this is how i compose imageMesage in protobuf before it encrypted and sent.

def toProtobufMessage(self):
    mediaKey=binascii.unhexlify(mediaKey)    --> mediaKey we genereted when uploading enc image
    fileHash= filehash      --> filehash we got as in responsed when uploading enc image
    url=self.url..encode('latin-1') -> url we got in responsed when uploading enc image
    image_message = ImageMessage()
    image_message.url = url
    image_message.width = self.width
    image_message.height = self.height
    image_message.mime_type = self.mimeType
    image_message.file_sha256 = self.fileHash
    image_message.file_length = self.size
    image_message.caption = "judul"
    image_message.jpeg_thumbnail = self.preview
    image_message.media_key = mediaKey

    return image_message

i still doubt in filehash (because when i observed when receiving image in binary format, but i sent still in plain hash).

So if i put [DATA] as random string, the message will reached the destination phone, but we no image only text: Waiting for this message. This may take a while. Learn More

But if i put [DATA] with the encrypted ImageMessage. the message is sent ang i got ACK but the message didn't reached the destination phone.

Currently i still didn't have idle time for further analyzing. May be if somebody can sniff from whatsapp web how is the propper node structure to send image message, will help to speedup solving this case.

[bahtiarp]

i tried to send image from yowsup A to yowsup B, in B logs i found that the message arrived but, yowsup cannot parsed the enc message.

File "/home/dev/WA/py/yowsup-master/yowsup/layers/axolotl/layer_receive.py", line 137, in handleWhisperMessage whisperMessage = WhisperMessage(serialized=enc.getData()) File "/usr/lib/python2.7/site-packages/axolotl/protocol/whispermessage.py", line 38, in init whisperMessage.ParseFromString(message) google.protobuf.message.DecodeError: Error parsing message.

[duzzifelipe]

@bahtiarp maybe the way you send is leaving the "enc" node child empty and decoder cannot understand.

And did you get to send an image message? I'm almost there. The closest I got was this:
I know I forgot something, but I didn't find it yet.

[bahtiarp]

if i fill "enc" child node empty, the message reached the dest, but no image warning message only. if i fill with encrypted imageMessage protobuf, then i got ack but the image didn't rendered on dest. (i know that the message is reached the dest, but still not proper so the dest cannot render).

btw, can you show your send log when the data shown as in your screen captured. i'm affraid that the enc attribute not attribute for image type.

[bahtiarp]

my new progress, i can send image from yowsup A to yowsup B. and message and image decrypted properly, and i can view the image correctly. but if i send from yowsup to whatsapp phone the image still not rendered although , my yowsup get

• [ack]
• and [receipt]

[bahtiarp]

The prev image now come but cannot renderd the full image. when click error message: Download failed "The download was unable to complete. Please try again later."

[duzzifelipe]

Maybe the problem now is with correctly upload encrypted media, because the preview is generated based on the file path (at least on my repo) and does not depends on the media upload.

This week I had no time to do it right and I even lost 2 numbers. But I'll try this weekend, at least get at the same step as you.

[dietzi]

@bahtiarp how did you got to this point? Can you send a diff please? So we can work together to get the problem solved. I'm sniffing the packets by using a proxy on my home-server, so I can decrypt the SSL-messages.

[dietzi]

I tried with sniffing but WhatsApp (on Android) expects a special certificate for SSL-connection. So my fakecert wont't work.....

[bahtiarp]

@dietzi , this is the diff: diff

with that i now can:

• send image from yowsup to yowsup
• send image from yowsup to whatsapp phone only image preview appeared
• send image (images that already in the wa phone before) from yowsup to whatsapp will appeared correctly.

now 2.10 am morning, need for sleep now 😴

[dietzi]

Your link don't work. Good night ;)