Something wrong with PUSH_UP, ACK

[JEBAHUMA]

After POC Offchain, the miners don't send the beacons through the middlmen.

[simeononsecurity]

@simeononsecurity

yes the pull_ack and push_ack suppose to go to mines as well, yes.

Update available on my repo.

    # Handle PULL_ACK message
    def handle_PULL_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for vgw in self.vgateways_by_mac.values():
            vgw_address = (vgw.server_ip, vgw.port)
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PULL_ACK has been received
        self.vgw_logger.debug(f"PULL_ACK received from gateway at {msg['MAC']}")

    # Handle PUSH_ACK message
    def handle_PUSH_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for vgw in self.vgateways_by_mac.values():
            vgw_address = (vgw.server_ip, vgw.port)
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PUSH_ACK has been received
        self.vgw_logger.debug(f"PUSH_ACK received from packet forwarder at {msg.get('MAC', addr)}")

[simeononsecurity]

And I broke it again. hold on

[simeononsecurity]

@powerthesa

Ok try again

    # Handle TX_ACK message
    def handle_TX_ACK(self, msg, addr):
        # Extract the token from the message
        token = msg['token']
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Update the JSON data with the correct token
        json_data = None
        if len(msg) > 12:
            json_data = msg[12:]
            json_obj = json.loads(json_data)
            json_obj['token'] = token
            json_data = json.dumps(json_obj).encode('utf-8')
        # Encode the message with the updated JSON data and send it back to all the virtual gateways
        for addr, vgw in self.vgateways_by_addr.items():
            mac_address = self.vgateways_by_mac[vgw.mac].mac
            vgw_address = (addr[0], addr[1])
            rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac_address, 'data': json_data})
            self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
            self.sock.sendto(rawmsg, vgw_address)
            # Check the error field in the JSON object to determine if the downlink request was accepted or rejected
            if json_data:
                json_obj = json.loads(json_data)
                error = json_obj.get('txpk_ack', {}).get('error', 'NONE')
                if error == 'NONE':
                    # Log a debug message indicating that the downlink request was accepted
                    self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")
                else:
                    # Log a debug message indicating that the downlink request was rejected
                    self.vgw_logger.debug(f"Downlink request rejected by gateway at {vgw_address}: {error}")
            else:
                # Log a debug message indicating that the downlink request was accepted
                self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")

    # Handle PULL_ACK message
    def handle_PULL_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for addr, vgw in self.vgateways_by_addr.items():
            vgw_address = (addr[0], addr[1])
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PULL_ACK has been received
        self.vgw_logger.debug(f"PULL_ACK received from gateway at {msg.get('MAC', addr)}")

    # Handle PUSH_ACK message
    def handle_PUSH_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for addr, vgw in self.vgateways_by_addr.items():
            vgw_address = (addr[0], addr[1])
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PUSH_ACK has been received
        self.vgw_logger.debug(f"PUSH_ACK received from packet forwarder at {msg.get('MAC', addr)}")

https://github.com/simeononsecurity/helium-DIY-middleman/tree/dev

[JEBAHUMA]

@simeononsecurity runing now, let see what happen when a tx_ack is received.

[simeononsecurity]

Take 231337 lol

Updated again. Caught another error. Fixed it. All variables now resolve I believe.

Try again now @powerthesa

[JEBAHUMA]

@simeononsecurity ok, get the tx_ack.

Error:

-   File "/home/middleman/gateways2miners.py", line 435, in <module>
-     main()
-   File "/home/middleman/gateways2miners.py", line 429, in main
-     gw2miner.run()
-   File "/home/middleman/gateways2miners.py", line 124, in run
-     self.handle_TX_ACK(msg, addr)
-   File "/home/middleman/gateways2miners.py", line 272, in handle_TX_ACK
-     rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac, 'data': json_data})
-   File "/home/middleman/src/messages.py", line 182, in encode_message
-     rawmsg = msg_obj.encode(message_object)
-  File "/home/middleman/src/messages.py", line 139, in encode
-    super().encode(message_object)
-   File "/home/middleman/src/messages.py", line 41, in encode
-     self.data = struct.pack("=BHB", message_object.get('ver', 2), message_object.get('token'), message_object.get('identifier'))
- struct.error: required argument is not an integer

[simeononsecurity]

@simeononsecurity ok, get the tx_ack.

Error:

-   File "/home/middleman/gateways2miners.py", line 435, in <module>
-     main()
-   File "/home/middleman/gateways2miners.py", line 429, in main
-     gw2miner.run()
-   File "/home/middleman/gateways2miners.py", line 124, in run
-     self.handle_TX_ACK(msg, addr)
-   File "/home/middleman/gateways2miners.py", line 272, in handle_TX_ACK
-     rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac, 'data': json_data})
-   File "/home/middleman/src/messages.py", line 182, in encode_message
-     rawmsg = msg_obj.encode(message_object)
-  File "/home/middleman/src/messages.py", line 139, in encode
-    super().encode(message_object)
-   File "/home/middleman/src/messages.py", line 41, in encode
-     self.data = struct.pack("=BHB", message_object.get('ver', 2), message_object.get('token'), message_object.get('identifier'))
- struct.error: required argument is not an integer

Okay that's good. I can work with this I think. I'm going AFK for an hour or two and I will look at fixing this and let you know.

[simeononsecurity]

@simeononsecurity ok, get the tx_ack.

Error:

-   File "/home/middleman/gateways2miners.py", line 435, in <module>
-     main()
-   File "/home/middleman/gateways2miners.py", line 429, in main
-     gw2miner.run()
-   File "/home/middleman/gateways2miners.py", line 124, in run
-     self.handle_TX_ACK(msg, addr)
-   File "/home/middleman/gateways2miners.py", line 272, in handle_TX_ACK
-     rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac, 'data': json_data})
-   File "/home/middleman/src/messages.py", line 182, in encode_message
-     rawmsg = msg_obj.encode(message_object)
-  File "/home/middleman/src/messages.py", line 139, in encode
-    super().encode(message_object)
-   File "/home/middleman/src/messages.py", line 41, in encode
-     self.data = struct.pack("=BHB", message_object.get('ver', 2), message_object.get('token'), message_object.get('identifier'))
- struct.error: required argument is not an integer

Updated the code on the dev branch again for testing. It's going to be difficult to get this working. Essentially I'm having to fake the TX_ACK as well. The way we are handling tokens in this is completely random

token=random.randint(0, 2**16 - 1)

According to https://github.com/Lora-net/packet_forwarder/blob/master/PROTOCOL.TXT The TX_ACK needs to have the same Token as the PULL_RESP. I wasn't able to figure out how to do that exactly. All the handlers in this use a random token so.. So I'm also randomly assigning the token for TX_ACK.

[JEBAHUMA]

@simeononsecurity nop, nothing happen. I'm done with that, i spent a lot of time trying to fix it. maybe i will have a look when i have more time.

[simeononsecurity]

Well at least the code is working ish. What we need to do really is set it up to where the PULL_RESP doesn't include fake data and have it repond with the TX_ACK only to the appropriate device and not all devices. The TX_ACK is an acknowledgement. It shouldn't contain any actual data. It's basically just a handshake.

[simeononsecurity]

Updated the dev branch again. I need more testers.

[simeononsecurity]

Merged my dev into main. Best I can tell my code is working. YMMV.

https://github.com/simeononsecurity/helium-DIY-middleman/tree/master

[sblanchard]

I’ll give feedback on testing. Last version crashed without error in log and the service restarted every few packets (so it seemed)

[JEBAHUMA]

@simeononsecurity i just had some time to do some test. and first we have to solve the first problem i see. to go to the next

The

for addr, vgw in self.vgateways_by_addr.items():
            mac_address = self.vgateways_by_mac[vgw.mac].mac
            vgw_address = (addr[0], addr[1])

is a loop of all vgateways (ip, port) and is sending the same ack to all the miners instead of sending the ack to only to the miner form we get the ack.

if we solve this, i can handl to see the next issue we if we get any. i think we are so near to solve this.

[Joehoehoepi]

I will test the new revision and come back to you guys, thanks for trying to resolve this

[simeononsecurity]

@simeononsecurity i just had some time to do some test. and first we have to solve the first problem i see. to go to the next

The

for addr, vgw in self.vgateways_by_addr.items():
            mac_address = self.vgateways_by_mac[vgw.mac].mac
            vgw_address = (addr[0], addr[1])

is a loop of all vgateways (ip, port) and is sending the same ack to all the miners instead of sending the ack to only to the miner form we get the ack.

if we solve this, i can handl to see the next issue we if we get any. i think we are so near to solve this.

Good catch didn't remember to fix that. I've updated the dev branch of my repo. Please confirm.

[simeononsecurity]

If anyone has gateway-rs logs available for the PULL_RESP - TX_ACK chain that went through the MM software please send them to me.

[simeononsecurity]

also please use -d for debug while executing so you see the debug statements before it errors out if at all.

[Joehoehoepi]

MM runs without any errors over here

[simeononsecurity]

You'll need to wait for a pull response to happen and it'll debug the tx_ack as part of that

[proxynetul]

@Joehoehoepi @JEBAHUMA any news guys ? is working for you ?

[sblanchard]

Only for witnessing but not beaconing

[proxynetul]

Beaconing is important

[Joehoehoepi]

same here no beacons only witnessing

[simeononsecurity]

@Joehoehoepi @JEBAHUMA any news guys ? is working for you ?

I've dropped interest in the fix. Currently my repo lays some of the groundwork for a future fix. But it'll require almost a complete rewrite of the software to get it working. The fix was beyond my skill. But it doesn't mean it won't work again eventually. It's just not worth beating my head against the desk for the coming months to get it working again. I already dropped like 40 hours into it.

[Joehoehoepi]

@simeononsecurity No progress for me, there is also a lut issue going on atm, but I don't get the tx power error so the tx_acks are not comming through. Strange thing is that when I leave out the mitm and just connect straight to "birdsound" ;) it is still not receiving a tx_ack or creating an tx power error. the problem could very well also be in the pktfwder. None the less thanks for the time you put in, I wish I was a better coder and that I could help you out more.