Something wrong with PUSH_UP, ACK

[JEBAHUMA]

After POC Offchain, the miners don't send the beacons through the middlmen.

[jibjab99]

How many miners are you running this with? I found the solution.

[JEBAHUMA]

what solution you found? i tried everything but without success.

[sblanchard]

@jibjab99 does your pr include the solution, I copied all your changes and reinstalled it but no luck.

[sblanchard]

possibly the way we identify beacons is not correct anymore rxpk.get('size') == 52 and rxpk.get('datr') == 'SF9BW125'

[JEBAHUMA]

That part is the poc challenge, not for sending beacons data i guess

[sblanchard]

Name of helium packet forwarder commit "Update txpk_ack buffer size to accomodate new material"

[JEBAHUMA]

with your modification is working now?

[sblanchard]

How many miners are you running this with? I found the solution.

You have the solution?

[proxynetul]

someone fixed it ?

[simeononsecurity]

Seems we have a bunch of undesirables here. If you don't have something to contribute, keep off this issue.

@jibjab99 the fork you're requesting to merge isn't that updated from the original. This repo is almost entirely unmaintained at this point. If you want to test curiousforkers repo, switch to it entirely. If you want to test it inside of a docker container. use mine. I've made modifications that randomize the SNR even better as well as a few other improvements.

I'm looking into the situation to see if I can solve it as well. However it's looking like it isn't so straight forward.

Also, if anyone has a good grasp on the gateway.rs changes and or is pretty good with rust. Message me.

[simeononsecurity]

https://github.com/simeononsecurity/helium-DIY-middleman

docker run \
    --net='bridge' \
    -p 1681:1681/udp \
    -e middleman_port=1681 \
    -e middleman_tx_adjust='--tx-adjust 0' \
    -e middleman_rx_adjust='--rx-adjust 0' \
    -e gateway_ID=AA555A0000000000 \
    -e server_address=localhost \
    -e serv_port_up=1680 \
    -e serv_port_down=1680 \
    --name diymiddleman -P -td simeononsecurity/helium_diy_middleman:latest

[JEBAHUMA]

i think i got where is the problem, but i don't have the skills to do it. the problem i think is that the middlman don't send a non-empty txpk_ack (Like: txpk_ack={"error": "NONE"}) to the miner to confirm the succesufull transmition of the fake push_data.

[simeononsecurity]

i think i got where is the problem, but i don't have the skills to do it. the problem i think is that the middlman don't send a non-empty txpk_ack (Like: txpk_ack={"error": "NONE"}) to the miner to confirm the succesufull transmition of the fake push_data.

I've already got a fix for this untested if it solves the issue in it's entirety

It's in my github repo

    def get_stat(self):
        """
        return data, address where data is raw bytearray to send from socket and address is the destination (port, ip)
        if no message should be sent returns None, None
        :return:
        """
        payload = dict(
            stat=dict(
                time=dt.datetime.utcnow().isoformat()[:19] + " GMT",
                rxnb=self.rxnb,
                rxok=self.rxnb,
                rxfw=self.rxnb,
                txnb=self.txnb,
                dwnb=self.txnb,
                ackr=100.0
            )
        )
        return self.__get_PUSH_DATA__(payload)

    def get_rxpks(self, msg):
        new_rxpks = []

        # next iterate through each received packet to see if it is a repeat from cached
        for rx in msg['data']['rxpk']:

            # modify metadata as needed
            modified_rx = self.rxmodifier.modify_rxpk(rx, src_mac=msg['MAC'], dest_mac=self.mac)

            # add rx payload to array to be sent to miner
            new_rxpks.append(modified_rx)

        if not new_rxpks:
            return None, None
        payload = dict(rxpk=new_rxpks)

        self.rxnb += len(new_rxpks)
        status, message = self.__get_PUSH_DATA__(payload)
        if status == 0:
            self.logger.debug(f"sending PUSH_DATA with {len(new_rxpks)} packets from vGW:{self.mac[-8:]} to miner {(self.server_address, self.port_up)}")
            return None, None
        else:
            self.logger.error(f"error sending PUSH_DATA to miner {(self.server_address, self.port_up)}: {message}")
            return message, None

    def __get_PUSH_DATA__(self, payload):
        """
        Sends PUSH_DATA message to miner with payload contents
        :param payload: raw payload
        :return: (status, message) where status is 0 for success and 1 for failure, and message is None if successful, error message otherwise
        """
        try:
            top = dict(
                _NAME_=MsgPushData.NAME,
                identifier=MsgPushData.IDENT,
                ver=2,
                token=random.randint(0, 2**16-1),
                MAC=self.mac,
                data=payload
            )
            payload_raw = encode_message(top)
            # send the message
            # ...
            # return success status and None for message
            return (0, None)
        except Exception as e:
            # return failure status and error message
            return (1, str(e))

[JEBAHUMA]

get [ERROR ] Error sending PULL_DATA message: 'VirtualGateway' object has no attribute 'socket' comes from vgateway.py

[simeononsecurity]

Appologies. I see the error now. I'm unable to do the level of testing that I need for this currently. (I'm no where near the hardware) I'm pushing an update now. Remove the docker container and it's image and try again in about 15 minutes or whenever this automated build is done https://github.com/simeononsecurity/helium-DIY-middleman/actions/runs/4159374466/jobs/7195372510

[JEBAHUMA]

no worries, i'm not using a docker for the moment to do fast testing, when it's solved i will run a docker then.

[simeononsecurity]

Roger. Well the code is updated. Just gotta pull it from the repo then. Also fyi, docker is just way better. You should give it a try.

[simeononsecurity]

I reverted the changes. Ended up causing more problems than fixing.

[JEBAHUMA]

I see, with last update (before the reveral) the Error sending PULL_DATA message: 'VirtualGateway' object has no attribute 'socket' still there.

[simeononsecurity]

So if anyone has any bright ideas I can turn it into the code. I'm not sure I have a grasp on it just yet.

The middle man software currently generates a fake tx ack. I was led to believe that is the issue but I'm starting to think that isn't so.

[JEBAHUMA]

is generating a empty tx ack? is has to be a non empty ack if not, will not work i guess.

[simeononsecurity]

is generating a empty tx ack? is has to be a non empty ack if not, will not work i guess.

I beleive it omits it even if there are errors. This is the documentation for how semtech handles it https://github.com/Lora-net/packet_forwarder/blob/d0226eae6e7b6bbaec6117d0d2372bf17819c438/PROTOCOL.TXT#L404

[JEBAHUMA]

i see, but i don't see where is the code when is pushing the fake tx ack to the miners when the middleman receive a downstream from the miners

[simeononsecurity]

I've looked into this further. It wasn't the tx_ack that was being faked. I was able to increase verbosity quite a bit. However, fixing this is going to require a level of refactoring I do not have the skills or time for. I added some comments and some new debugging output in my repo. Feel free to take a look. But I'm not gonna spend anymore time trying to fixing it. https://github.com/simeononsecurity/helium-DIY-middleman

[JEBAHUMA]

@simeononsecurity thank you.

[simeononsecurity]

To justify the 10 plus hours I spent reversing this repo, I went back and added a bunch of useful comments explaining what things do in my repo. Hopefully this makes it easier for the next guy. But definitely over my head.

Just updated. https://github.com/simeononsecurity/helium-DIY-middleman

[jibjab99]

"It wasn't the tx_ack that was being faked."

It's not being faked, it's just not being sent.

[simeononsecurity]

"It wasn't the tx_ack that was being faked."

It's not being faked, it's just not being sent.

Reguardless of which. How have you determined this is the issue with the beacons? No one can tell me. When reviewing the gateway-rs code it doesn't look like it even really matters if it is acknowledged or not.

[jibjab99]

It is 100% the ACK.

If you care to continue looking at it you can use util_tx_continuous to test sending data to the gateway.

You are just sending an empty TX_ACK of 12 bytes back to the miner.

[JEBAHUMA]

@simeononsecurity cause the new POC offchain required a tx_ack to submit the beacon, it was already mention by the developers. i have debug all from your new update and efectively the miner send a tx_ack to the middlmen. but the middleman never send back the TX_ACK to the miner to confirm.

TX_ACK received from XXXXXX. Data: {'ver': 2, 'token': XXXX, 'identifier': 5, '_NAME_': 'TX_ACK', '_UNIX_TS_': XXXXXXXXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXXX}}}"

[simeononsecurity]

@simeononsecurity cause the new POC offchain required a tx_ack to submit the beacon, it was already mention by the developers. i have debug all from your new update and efectively the miner send a tx_ack to the middlmen. but the middleman never send back the TX_ACK to the miner to confirm.

TX_ACK received from XXXXXX. Data: {'ver': 2, 'token': XXXX, 'identifier': 5, '_NAME_': 'TX_ACK', '_UNIX_TS_': 1676334512.9704683, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXXX}}}"

Where is this information coming from

Data: {'ver': 2, 'token': XXXX, 'identifier': 5, '_NAME_': 'TX_ACK', '_UNIX_TS_': 1676334512.9704683, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXXX}}}"

If I could get that reliably passed into this I could forward the data..

    # Handle TX_ACK message
    def handle_TX_ACK(self, msg, addr=None): 
        # Get the mac address from the message or the address
        mac_address = msg.get('mac', addr)
        # Log a debug message indicating that a TX_ACK has been received
        self.vgw_logger.debug(f"TX_ACK received from {mac_address}")

[simeononsecurity]

Cause my def here is what is causing the TX_ACK received from XXXXXX output But I couldn't figure out how to get the correct data to pass to the miners.

[simeononsecurity]

It is 100% the ACK.

If you care to continue looking at it you can use util_tx_continuous to test sending data to the gateway.

You are just sending an empty TX_ACK of 12 bytes back to the miner.

There isn't an already compiled release of this. I compiled the util_tx_contentious

sos@testbox:~/lora_gateway/util_tx_continuous$ ./util_tx_continuous -f 868 -r 1257 --dig 0 --mix 14 --pa 3 --mod "LORA" --sf 7 --bw 125
ERROR: failed to start the concentrator

I'm assuming I have to run this on a lora gateway itself... Correct?

[JEBAHUMA]

@simeononsecurity i just put your

# Handle TX_ACK message
    def handle_TX_ACK(self, msg, addr=None): 
        # Get the mac address from the message or the address
        mac_address = msg.get('mac', addr)
        # Log a debug message indicating that a TX_ACK has been received
        self.vgw_logger.debug(f"TX_ACK received from {mac_address}")

and i receive the debug when the miner send the tx_ack and the result is this, obviously the XXXXXXXX are filled with my miner data.

[simeononsecurity]

Roger. Well then someone tell me more about how to get this continuous tx setup so I can test this?

[simeononsecurity]

@jibjab99 Is there another utility perhaps we could try? I'm unable to get that compiled on any of my lora devices. Ones I have root access to don't have enough storage to install GCC to compile them and I'm not about to figure out how to cross compile tonight lol.

[simeononsecurity]

I went down that rabbit hole. Looks like that only supports the fully diy hotspots anyways. I don't have any of those..

[simeononsecurity]

@jibjab99 @powerthesa

I wasn't able to test entirely this code. But it works from what I gather. Everything is tested except the TX_ACK part https://github.com/simeononsecurity/helium-DIY-middleman/tree/dev The thing I don't know is if specifying addr for the self.socket.sendto is enough. The way I read it it seems I need the MAC address and not the IP. But what do I know... I'm not sure how you're getting the json output in addition to things and able to get the mac. But if you can run this change and show me the output from your end that would be great!

    # Handle TX_ACK message
    def handle_TX_ACK(self, msg, addr):
        #print(messages.MsgTxAck.decode(self.get_message))
        # Extract the token and MAC address from the message
        token = msg[1:3]
        mac_address = msg.get('MAC', addr)
        rawmsg = messages.encode_message(msg)

        # Extract the optional JSON object from the message
        json_data = None
        if len(msg) > 12:
            json_data = msg[12:]

        # Check the error field in the JSON object to determine if the downlink request was accepted or rejected
        if json_data:
            json_obj = json.loads(json_data)
            error = json_obj.get('txpk_ack', {}).get('error', 'NONE')
            if error == 'NONE':
                # Log a debug message indicating that the downlink request was accepted
                self.vgw_logger.debug(f"Downlink request accepted by gateway at {mac_address}")
                self.sock.sendto(rawmsg, addr)
                self.vgw_logger.debug(f"Decoded Message: {msg}")
                self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
            else:
                # Log a debug message indicating that the downlink request was rejected
                self.vgw_logger.debug(f"Downlink request rejected by gateway at {mac_address}: {error}")
                self.sock.sendto(rawmsg, addr)
                self.vgw_logger.debug(f"Decoded Message: {msg}")
                self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
        else:
            # Log a debug message indicating that the downlink request was accepted
            self.vgw_logger.debug(f"Downlink request accepted by gateway at {mac_address}")
            self.sock.sendto(rawmsg, addr)
            self.vgw_logger.debug(f"Decoded Message: {msg}")
            self.vgw_logger.debug(f"Encoded Message: {rawmsg}")

    # Handle PULL_ACK message
    def handle_PULL_ACK(self, msg, addr):
        rawmsg = messages.encode_message(msg)
        self.sock.sendto(rawmsg, addr)
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
        # Extract the mac address from the message
        mac_address = msg.get('MAC', addr)
        # Log a debug message indicating that a PULL_ACK has been received
        self.vgw_logger.debug(f"PULL_ACK received from gateway at {mac_address}")

    # Handle PUSH_ACK message
    def handle_PUSH_ACK(self, msg, addr):
        rawmsg = messages.encode_message(msg)
        self.sock.sendto(rawmsg, addr)
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
        # Get the mac address from the message or the address
        mac_address = msg.get('MAC', addr)
        # Log a debug message indicating that a PUSH_ACK has been received
        self.vgw_logger.debug(f"PUSH_ACK received from packet forwarder at {mac_address}")

[JEBAHUMA]

Getting error:

in handle_TX_ACK token = msg[1:3] TypeError: unhashable type: 'slice'

[JEBAHUMA]

@jibjab99 you said that you found the solution for the ack, Could you please shed some light on the matter?

[simeononsecurity]

Getting error:

in handle_TX_ACK token = msg[1:3] TypeError: unhashable type: 'slice'

Roger I've updated to remove the code but output a few things. Can you try again and give me the output it gives when you get a TX_ACK?

[JEBAHUMA]

@simeononsecurity i don't see the update. But i did some test changing some lines of your code. and adding to the debug to show the data as i did in the past as with self.vgw_logger.debug(f"TX_ACK received from {mac_address} Data: {msg}")

here is the debug of the tx_ack:

TX_ACK received from ('10.12.1.23', 35380). Data: {'ver': 2, 'token': XXXXXX, 'identifier': 5, '_NAME_': 'TX_ACK', '_UNIX_TS_': XXXXXXXX.XXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXX}}}

Decoded Message: {'ver': 2, 'token': XXXXXX, 'identifier': 5, '_NAME_': 'TX_ACK', '_UNIX_TS_': XXXXXXXX.XXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXX}}}

Encoded Message: b'\x022\xc7\xq5\xa2_\x15\sdf\xgqh4\x85{"txpk_ack": {"error": "NONE", "tmst": XXXXXXX}}'

As you asked before, the tx_ack has to be sent to an ip, the code take an ip adress, but this ip adress i thing with this code is sending the tx_ack to the concentrator and not to the miner. as is taken the ip adress of the gateway and not of the vgateway. btw this can be fixed only by forcing the port to be one of the vgateway up/down ports.

[simeononsecurity]

@simeononsecurity i don't see the update. But i did some test changing some lines of your code. and adding to the debug to show the data as i did in the past as with self.vgw_logger.debug(f"TX_ACK received from {mac_address} Data: {msg}")

here is the debug of the tx_ack:

TX_ACK received from ('10.12.1.23', 35380). Data: {'ver': 2, 'token': XXXXXX, 'identifier': 5, '_NAME_': 'TX_ACK', '_UNIX_TS_': XXXXXXXX.XXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXX}}}

Decoded Message: {'ver': 2, 'token': XXXXXX, 'identifier': 5, '_NAME_': 'TX_ACK', '_UNIX_TS_': XXXXXXXX.XXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXX}}}

Encoded Message: b'\x022\xc7\xq5\xa2_\x15\sdf\xgqh4\x85{"txpk_ack": {"error": "NONE", "tmst": XXXXXXX}}'

As you asked before, the tx_ack has to be sent to an ip, the code take an ip adress, but this ip adress i thing with this code is sending the tx_ack to the concentrator and not to the miner. as is taken the ip adress of the gateway and not of the vgateway. btw this can be fixed only to force the port to be one of the vgateway up/down ports.

It's on the dev branch so I don't break any of my auto updating docker containers in the process.. Thanks for that. I think I know what to do now. Lets hope. Fingers crossed

[simeononsecurity]

@powerthesa

Should the handle_PULL_ACK and handle_PUSH_ACK go to the miner or to the vgateway? Also to confirm, should the handle_TX_ACK go to the vgateway?

Cause the way I understood the TX_ACK it should go to the miner no? https://github.com/Lora-net/packet_forwarder/blob/master/PROTOCOL.TXT

I may be confused. So by sending it to the middleman vgateway it should end up at the miner... Am I safe to assume this?

[JEBAHUMA]

@simeononsecurity yes, the handle_tx_ack should go to the miner (vgateway).

[JEBAHUMA]

@simeononsecurity

let's assume you have:

added vgateway for miner at 192.168.1.2 port: 1680(up)/1680(dn)

You receive a TX_ACK from the packet forwarder => ('192.168.1.2', 35380). withe Data: {'ver': 2, 'token': XXXXXX, 'identifier': 5, 'NAME': 'TX_ACK', '_UNIXTS': XXXXXXXX.XXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXX}}}

So the TX_ACK should go to the miner => ('192.168.1.2', 1680)

[simeononsecurity]

@simeononsecurity

let's assume you have:

added vgateway for miner at 192.168.1.2 port: 1680(up)/1680(dn)

You receive a TX_ACK from the packet forwarder => ('192.168.1.2', 35380). withe Data: {'ver': 2, 'token': XXXXXX, 'identifier': 5, 'NAME': 'TX_ACK', '_UNIXTS': XXXXXXXX.XXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXX}}}

So the TX_ACK should go to the miner => ('192.168.1.2', 1680)

Isn't my best work, but I think this should work..

    # Handle TX_ACK message
    def handle_TX_ACK(self, msg, addr):
        # Extract the token from the message
        token = msg['token']
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Update the JSON data with the correct token
        json_data = None
        if len(msg) > 12:
            json_data = msg[12:]
            json_obj = json.loads(json_data)
            json_obj['token'] = token
            json_data = json.dumps(json_obj).encode('utf-8')
        # Encode the message with the updated JSON data and send it back to all the virtual gateways
        for vgw in self.vgw.values():
            vgw_address = (vgw.ip, vgw.port)
            rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': vgw.mac, 'data': json_data})
            self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
            self.sock.sendto(rawmsg, vgw_address)
            # Check the error field in the JSON object to determine if the downlink request was accepted or rejected
            if json_data:
                json_obj = json.loads(json_data)
                error = json_obj.get('txpk_ack', {}).get('error', 'NONE')
                if error == 'NONE':
                    # Log a debug message indicating that the downlink request was accepted
                    self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")
                else:
                    # Log a debug message indicating that the downlink request was rejected
                    self.vgw_logger.debug(f"Downlink request rejected by gateway at {vgw_address}: {error}")
            else:
                # Log a debug message indicating that the downlink request was accepted
                self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")

It's updated on my dev branch

[simeononsecurity]

This is set up to forward to all the vgateways as well.

Also please confirm where the pull_ack and push_ack are supposed to go. I'm assuming the vgateway as well? @powerthesa

[simeononsecurity]

https://github.com/simeononsecurity/helium-DIY-middleman/tree/dev

[JEBAHUMA]

@simeononsecurity

yes the pull_ack and push_ack suppose to go to mines as well, yes.