search

nebulous / infinitude

Open control of Carrier/Bryant thermostats
MIT License
225 stars 50 forks source link

Serial bridge issues #138

Closed andyopengit closed 2 years ago

andyopengit commented 2 years ago

I recently migrated from running infinitude on a raspberry Pi located next to my HVAC unit to a docker based installation running with the rest of my home automation dockers. I was previously using a USB serial interface to read that information off the bus so wanted to continue to get that information to the docker.

I tried using ser2net on the Pi and sending that over to the docker using the SERIAL_SOCKET environment variable but was unable to get that data into the docker although I was able to open a telnet connection to the serial bridge service and see the data off the bus. I then purchased the serial bridge (https://www.amazon.com/gp/product/B09DCT1YDR) and am similarly able to see the data via telnet however the infinitude application still does not read the data.

I've tried removing the serial_tty line from the docker-compose file per a suggestion I read on another topic here but that does not appear to work either. Docker startup:

Creating infinitude ... done Attaching to infinitude infinitude | MODE is set to Development infinitude | Using 172.16.1.28:23 for serial interface infinitude | [2022-03-10 15:26:39.05099] [6] [info] Listening at "http://*:3000" infinitude | [2022-03-10 15:26:43.34144] [6] [info] weather-20817-forecast cached or passthru disabled infinitude | [2022-03-10 15:26:43.34178] [6] [info] /weather/20817/forecast infinitude | [2022-03-10 15:26:55.89972] [6] [info] systems-1519W101553-status cached or passthru disabled infinitude | [2022-03-10 15:26:55.90013] [6] [info] /systems/1519W101553/status infinitude | [2022-03-10 15:26:55.93025] [6] [info] Saving status infinitude | [2022-03-10 15:27:08.05543] [6] [info] systems-1519W101553-status cached or passthru disabled infinitude | [2022-03-10 15:27:08.05582] [6] [info] /systems/1519W101553/status infinitude | [2022-03-10 15:27:08.06413] [6] [info] Saving status infinitude | [2022-03-10 15:27:08.64012] [6] [info] Attempting to access 172.16.1.28:23 infinitude | Use of uninitialized value in substr at ./infinitude line 399.

docker-compose.yaml:

version: '3'

services: infinitude: container_name: infinitude hostname: infinitude image: scyto/infinitude

image: nebulous/infinitude:latest

# build:
#   context: .
#   dockerfile: Dockerfile
# network_mode: host
# ports:
#   - "8080:3000"
volumes:
  - pv_infinitude:/infinitude
  - /etc/localtime:/etc/localtime:ro
environment:
  - TZ=America/New_York
  - APP_SECRET=Pogotudinal
  - PASS_REQS=0
  - MODE=Development
  - SERIAL_SOCKET=172.16.1.28:23
# - SERIAL_TTY=null
# devices:
#   - /dev/ttyUSB0:/dev/ttyUSB0
networks:
  default:
    ipv4_address: 172.16.1.209
restart: unless-stopped

networks: default: external: name: macvlan0

volumes: pv_infinitude: external: true

infinitude.json file

{ "app_secret": "Pogotudinal", "pass_reqs": 300, "serial_socket": "172.16.1.28:23" }

Is there any way I might be able to test the perl module at the command line to make sure it's able to open the data from the serial bridge?

scyto commented 2 years ago

I just changed from my image to official nebulous one and my serial also just broke.... so not sure whats going on, will dig a little) (btw you don't need to use macvlan for infinitude, but no issues if you want to either) (oh also why are you mode development?)

mamarley commented 2 years ago

Did this start happening after https://github.com/nebulous/infinitude/commit/a290dd3c7f5aafa0fdde4214d5c4b713bc3abfaf? I did some shuffling around of how the controllers were loaded there to prevent them from loading multiple times, which could potentially have resulted in issues relating to serial. (I'm not able to test myself as I don't have an RS-485 adapter hooked up.)

nebulous commented 2 years ago

Recently I've been using a usb serial adaptor, but I hooked up an esp8266 bridge this morning and it seems to still be working with the latest infinitude docker build (on an x86-based Atomic pi). Are you able to telnet or nc to your serial bridge from inside the infinitude container?

something like the following

docker exec -it infinitude bash # get a bash prompt inside the running infinitude container
apt-get update # pull package list
apt-get install netcat -y # install netcat
nc TheIPofYourSerialBridge 23
andyopengit commented 2 years ago

Thanks all for getting back to me. I am able to both telnet and nc from within the container and receive data from the same IP and port configured in the config file. Any code I should/can comment out in the infinitude file to help isolate this further?

The docker image always seems to overwrite the config file but I assume this is known as it's always writing in the config from the environment variables passed in by docker.

PS - It would be nice to be able to pass in the port the service runs on as an environment variable.

`root@infinitude:/infinitude# nc 172.16.1.28 23 ?@y@  n. @x 

m~ @{VARIABLE SPEED FURNACE CESR131671-03 59MN7B080C2111203719M250030 4619A46597@;R  2 R{VAR SPD COMP VERSIONCESR131564-1024VNA948A0030050 441908290814319E10731 WF@   M   ssQN 0K E r`

@scyto - Apologies for not responding earlier. I am in development hoping there might be additional information given the troubleshooting. I currently run in macvlan because I'm lazy and just like to give each of my running containers their own IP. Planning on putting this behind trafik at some point in the future.

nebulous commented 2 years ago

The docker image always seems to overwrite the config file but I assume this is known as it's always writing in the config from the environment variables passed in by docker.

yep - the docker image expects to be self-contained/controlled from docker-compose/env

PS - It would be nice to be able to pass in the port the service runs on as an environment variable.

Port mapping should be done via docker ports eg

ports:
  - "3000:80"

I made some serial parsing changes a few months ago in preparation for doing device emulation on the bus (which I'll get to one of these years, but spare tuits have been scarce) which could plausibly be causing you trouble. I'm about to push one change to simplify things which may help, but in the mean time here's a throwaway script that you can run inside your infinitude container to test stream parsing with examples for files, tty, or tcp:

    ),
#!/usr/bin/perl
use strict;
use CarBus;
use Data::Dumper;
use IO::File;
use IO::Socket::IP;
use IO::Termios;

my $carbus = new CarBus(async=>1);
#my $sfh = new IO::File("somedumpfile.raw"); # dumpfile
#my $sfh = IO::Termios->open("/dev/ttyUSB0","38400,8,n,1"); #serial port
my $sfh = IO::Socket::IP->new(PeerHost=>'192.168.1.47', PeerPort=>23); #tcp

my $buffer = '';
while(1) {
        $sfh->recv($buffer, 128);
        $carbus->push_stream($buffer);
        my $frame = $carbus->get_frame();
        unless ($frame->{error}) {
                print Dumper($frame);
        }
}

Also, if it somehow helps, here's the docker-compose.yml file I'm using sucessfully with an esp8266 as a tcp bridge

version: "2.1"

services:
  infinitude:
    container_name: infinitude
    hostname: infinitude
    image: nebulous/infinitude:latest
    build:
      context: .
      dockerfile: Dockerfile
    network_mode: host
    ports:
      - "3000:3000"
    volumes:
      - ./state:/infinitude/state
    environment:
      - APP_SECRET=Pogotudinal
      - PASS_REQS=300
      - MODE=Production
#      - SERIAL_TTY=/dev/ttyUSB0
      - SERIAL_SOCKET=192.168.1.47:23
#    devices:
#      - /dev/ttyUSB0:/dev/ttyUSB0
    restart: always
andyopengit commented 2 years ago

I created a file "serial_test.sh" and put in the contents of your script added execute and tried to run. I seem to be getting an error that the CarBus module is not installed in the VM. I believe that is the module you wrote so I'm uncertain how I would go about installing that or adding to my path?

`root@infinitude:/infinitude# chmod +x serial_test.sh root@infinitude:/infinitude# ./serial_test.sh Can't locate CarBus.pm in @INC (you may need to install the CarBus module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl) at ./serial_test.sh line 3. BEGIN failed--compilation aborted at ./serial_test.sh line 3. root@infinitude:/infinitude# perl perl perl5.32-x86_64-linux-gnu perldoc perlthanks
perl5.32.1 perlbug perlivp
root@infinitude:/infinitude# perl serial_test.sh Can't locate CarBus.pm in @INC (you may need to install the CarBus module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.32.1 /usr/local/share/perl/5.32.1 /usr/lib/x86_64-linux-gnu/perl5/5.32 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl-base /usr/lib/x86_64-linux-gnu/perl/5.32 /usr/share/perl/5.32 /usr/local/lib/site_perl) at serial_test.sh line 3. BEGIN failed--compilation aborted at serial_test.sh line 3. root@infinitude:/infinitude# cat serial_test.sh

!/usr/bin/perl

use strict; use CarBus; use Data::Dumper; use IO::File; use IO::Socket::IP; use IO::Termios;

my $carbus = new CarBus(async=>1);

my $sfh = new IO::File("somedumpfile.raw"); # dumpfile

my $sfh = IO::Termios->open("/dev/ttyUSB0","38400,8,n,1"); #serial port

my $sfh = IO::Socket::IP->new(PeerHost=>'192.168.1.47', PeerPort=>23); #tcp

my $buffer = ''; while(1) { $sfh->recv($buffer, 128); $carbus->push_stream($buffer); my $frame = $carbus->get_frame(); unless ($frame->{error}) { print Dumper($frame); } } root@infinitude:/infinitude#
'

andyopengit commented 2 years ago

Ok I added a line to your script to include the lib directory so that it could find CarBus. Had to lookup how to do that. I now get an error on line 17: 'root@infinitude:/infinitude# ./serial_test.sh Can't call method "recv" on an undefined value at ./serial_test.sh line 17. root@infinitude:/infinitude# '

This would be the first line in the while statement.

andyopengit commented 2 years ago

Apologies I'm an idiot. Once I also changed the IP to my bridge IP I started getting a bunch of JSON data output to the screen. Would you like that output?

Sorry for the barrage of posts...

nebulous commented 2 years ago

ok, good. No, I don't need the output, all that script does is dump raw parsed frames it finds in a stream. The output is for you to see if perl/CarBus is actually able to talk to your serial bridge successfully, which it seems like it must be.

The ci pipeline is currently building a new set of container images which should be available to pull in a few minutes. If you could pull those and retry we can continue debugging if necessary.

scyto commented 2 years ago

(which I'll get to one of these years, but spare tuits have been scarce)

Want me to send you my SAM (SYSTXCCSAM01)? Its of no use to me now that I have the home assistant integration that works with infinitude?

scyto commented 2 years ago

I just pulled the new image from '43 mins ago' - no more errors in console output with mode-production

status page is more reliable, still having issues loading other pages reliably status looks good, comfort profiles is empty and so is schedules, about finally loaded but took a long time, serial isn't doing anything yet....

in chrome (edge) i am seeing this in the console occasionally - i do note in firefox the pages remain snappy and have a little more loaded - but still no data,

12ad8ede.vendor.js:5 

       TypeError: Cannot read properties of undefined (reading '0')
    at 5dfc4da2.scripts.js:1:1820
    at i (12ad8ede.vendor.js:6:5439)
    at 12ad8ede.vendor.js:6:5885
    at n.$digest (12ad8ede.vendor.js:6:11744)
    at n.$apply (12ad8ede.vendor.js:6:13738)
    at g (12ad8ede.vendor.js:5:19791)
    at r (12ad8ede.vendor.js:5:22360)
    at XMLHttpRequest.w.onload (12ad8ede.vendor.js:5:22784) 'Possibly unhandled rejection: {}'

in firefox

TypeError: b.data[e] is undefined
    reloadData http://swarm:4000/scripts/5dfc4da2.scripts.js:1
    i http://swarm:4000/scripts/12ad8ede.vendor.js:6
    k http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $digest http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $apply http://swarm:4000/scripts/12ad8ede.vendor.js:6
    g http://swarm:4000/scripts/12ad8ede.vendor.js:5
    r http://swarm:4000/scripts/12ad8ede.vendor.js:5
    onload http://swarm:4000/scripts/12ad8ede.vendor.js:5
    Zb http://swarm:4000/scripts/12ad8ede.vendor.js:5
    w http://swarm:4000/scripts/12ad8ede.vendor.js:5
    j http://swarm:4000/scripts/12ad8ede.vendor.js:5
    i http://swarm:4000/scripts/12ad8ede.vendor.js:6
    k http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $digest http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $apply http://swarm:4000/scripts/12ad8ede.vendor.js:6
    g http://swarm:4000/scripts/12ad8ede.vendor.js:4
    g http://swarm:4000/scripts/12ad8ede.vendor.js:4
    g http://swarm:4000/scripts/12ad8ede.vendor.js:4
    la http://swarm:4000/scripts/12ad8ede.vendor.js:4
    ka http://swarm:4000/scripts/12ad8ede.vendor.js:4
    <anonymous> http://swarm:4000/scripts/12ad8ede.vendor.js:9
    l http://swarm:4000/scripts/12ad8ede.vendor.js:2
    m http://swarm:4000/scripts/12ad8ede.vendor.js:2
    setTimeout handler*Deferred/then/f/< http://swarm:4000/scripts/12ad8ede.vendor.js:2
    k http://swarm:4000/scripts/12ad8ede.vendor.js:2
    fireWith http://swarm:4000/scripts/12ad8ede.vendor.js:2
    fire http://swarm:4000/scripts/12ad8ede.vendor.js:2
    k http://swarm:4000/scripts/12ad8ede.vendor.js:2
    fireWith http://swarm:4000/scripts/12ad8ede.vendor.js:2
    ready http://swarm:4000/scripts/12ad8ede.vendor.js:2
    m http://swarm:4000/scripts/12ad8ede.vendor.js:1
    EventListener.handleEvent* http://swarm:4000/scripts/12ad8ede.vendor.js:2
    <anonymous> http://swarm:4000/scripts/12ad8ede.vendor.js:1
    <anonymous> http://swarm:4000/scripts/12ad8ede.vendor.js:1
 Possibly unhandled rejection: {} 12ad8ede.vendor.js:5:31136
    e http://swarm:4000/scripts/12ad8ede.vendor.js:5
    get http://swarm:4000/scripts/12ad8ede.vendor.js:5
    j http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $digest http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $apply http://swarm:4000/scripts/12ad8ede.vendor.js:6
    g http://swarm:4000/scripts/12ad8ede.vendor.js:5
    r http://swarm:4000/scripts/12ad8ede.vendor.js:5
    onload http://swarm:4000/scripts/12ad8ede.vendor.js:5
    (Async: EventHandlerNonNull)
    Zb http://swarm:4000/scripts/12ad8ede.vendor.js:5
    w http://swarm:4000/scripts/12ad8ede.vendor.js:5
    j http://swarm:4000/scripts/12ad8ede.vendor.js:5
    i http://swarm:4000/scripts/12ad8ede.vendor.js:6
    k http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $digest http://swarm:4000/scripts/12ad8ede.vendor.js:6
    $apply http://swarm:4000/scripts/12ad8ede.vendor.js:6
    g http://swarm:4000/scripts/12ad8ede.vendor.js:4
    g http://swarm:4000/scripts/12ad8ede.vendor.js:4
    g http://swarm:4000/scripts/12ad8ede.vendor.js:4
    la http://swarm:4000/scripts/12ad8ede.vendor.js:4
    ka http://swarm:4000/scripts/12ad8ede.vendor.js:4
    <anonymous> http://swarm:4000/scripts/12ad8ede.vendor.js:9
    l http://swarm:4000/scripts/12ad8ede.vendor.js:2
    m http://swarm:4000/scripts/12ad8ede.vendor.js:2
    (Async: setTimeout handler)
    f http://swarm:4000/scripts/12ad8ede.vendor.js:2
    k http://swarm:4000/scripts/12ad8ede.vendor.js:2
    fireWith http://swarm:4000/scripts/12ad8ede.vendor.js:2
    fire http://swarm:4000/scripts/12ad8ede.vendor.js:2
    k http://swarm:4000/scripts/12ad8ede.vendor.js:2
    fireWith http://swarm:4000/scripts/12ad8ede.vendor.js:2
    ready http://swarm:4000/scripts/12ad8ede.vendor.js:2
    m http://swarm:4000/scripts/12ad8ede.vendor.js:1
    (Async: EventListener.handleEvent)
    <anonymous> http://swarm:4000/scripts/12ad8ede.vendor.js:2
    <anonymous> http://swarm:4000/scripts/12ad8ede.vendor.js:1
    <anonymous> http://swarm:4000/scripts/12ad8ede.vendor.js:1
scyto commented 2 years ago

also getting some of these in browser console when status fails to load i think

12ad8ede.vendor.js:5 

       Error: [ngModel:nonassign] Expression 'carbus.outsideTemp || status.oat[0]' is non-assignable. Element: <ng-dial-gauge ng-if="status.cfgem[0]=='F'" title="Outside" units="°" border-width="0" bar-color="#0000FF" bar-color-end="#FF2010" bar-width="10" ng-model="carbus.outsideTemp || status.oat[0]" class="ng-pristine ng-untouched ng-valid ng-scope ng-isolate-scope">
https://errors.angularjs.org/1.8.2/ngModel/nonassign?p0=carbus.outsideTemp%20%7C%7C%20status.oat%5B0%5D&p1=%3Cng-dial-gauge%20ng-if%3D%22status.cfgem%5B0%5D%3D%3D'F'%22%20title%3D%22Outside%22%20units%3D%22%C2%B0%22%20border-width%3D%220%22%20bar-color%3D%22%230000FF%22%20bar-color-end%3D%22%23FF2010%22%20bar-width%3D%2210%22%20ng-model%3D%22carbus.outsideTemp%20%7C%7C%20status.oat%5B0%5D%22%20class%3D%22ng-pristine%20ng-untouched%20ng-valid%20ng-scope%20ng-isolate-scope%22%3E
    at 12ad8ede.vendor.js:3:27190
    at Object.$$initGetterSetters (12ad8ede.vendor.js:8:27754)
    at Object.pre (12ad8ede.vendor.js:9:1001)
    at 12ad8ede.vendor.js:4:821
    at Ca (12ad8ede.vendor.js:5:10257)
    at o (12ad8ede.vendor.js:5:1679)
    at g (12ad8ede.vendor.js:4:29500)
    at 12ad8ede.vendor.js:4:29017
    at 12ad8ede.vendor.js:5:41
    at d (12ad8ede.vendor.js:4:30274) `<ng-dial-gauge ng-if="status.cfgem[0]=='F'" title="Outside" units="°" border-width="0" bar-color="#0000FF" bar-color-end="#FF2010" bar-width="10" ng-model="carbus.outsideTemp || status.oat[0]" class="ng-pristine ng-untouched ng-valid ng-scope ng-isolate-scope">`
andyopengit commented 2 years ago

I am seeing the same error in Chrome on Mac. I am still unable to see any serial information.

The docker logs are far quieter however that is likely related to the change to production mode.

FYI - I'm just outside DC myself on the Maryland side!

andyopengit commented 2 years ago

Whenever I attempt to load the serial page I get the following immediately in the docker log: 'infinitude | [2022-03-18 13:49:07.21449] [6] [info] Attempting to access 172.16.1.28:23 infinitude | [2022-03-18 13:49:07.24518] [6] [info] Closed: 1006 infinitude | [2022-03-18 13:49:07.24553] [6] [info] Closed stream'

scyto commented 2 years ago

sorry should have done this earlier, yes netcat works inside the container and sees serial output, just nothing on the web page.

nebulous commented 2 years ago

I can actually reproduce this issue, at least as @scyto sees it - apparently I was a dummy and needed to clear my browser cache. I can confirm that the backend is parsing streams from sockets or serial devices and shipping them out the websocket. The frontend is receiving said data via the websocket, but something is preventing the serial page from rendering/refreshing. @mamarley has the most recent contact with this code so he might have something to say about it - otherwise I'll see if I can pull an older hash to see if the problem is a frontend regression or something to do with the data shape changing subtly. @andyopengit - it looks like you're having a slightly different problem in that the backend isn't making or maintaining contact with your serial bridge. Hopefully I can get to the bottom of the rendering issue and we can further debug your connections.

Want me to send you my SAM (SYSTXCCSAM01)? Its of no use to me now that I have the home assistant integration that works with infinitude?

maybe so - though I wouldn't want to be gifted it since who knows how long it would take me to actually get time to work on emulation. Your dumps have already proven valuable to generate SAMlike frames on my bus - but I need to eventually get the bootup negotiation working so that the stat believes there's a SAM present.

scyto commented 2 years ago

though I wouldn't want to be gifted it since who knows how long it would take me to actually get time to work on emulation.

literally it would be yours, once you don't need it (years from now) we could revisit that but i don't know why i would want it back, lol. Since i moved to home assistant and the infinitude integration i have no use for it. PM me an address and I will send it, no strings attached.

mamarley commented 2 years ago

I strongly suspect this is due to the change of how the controllers are loaded in https://github.com/nebulous/infinitude/commit/a290dd3c7f5aafa0fdde4214d5c4b713bc3abfaf. I initially thought that the $rootScope stuff would continue to work even though the controller was only loaded for the on index.html:29, but that appears not to be the case.

I wonder if it might be a better idea, since both controllers are loaded for all tabs anyway, just to combine the SerialCtrl functionality into MainCtrl and then use $scope for everything instead of $rootScope?

EDIT: I've implemented that in https://github.com/mamarley/infinitude/tree/unify-controller if anyone wants to give it a shot. I can confirm it still works for everything except serial, but I don't have a system with an RS-485 adapter connected, so I can't test that.

nebulous commented 2 years ago

@mamarley yep - I pulled unify-controller and the problem disappears ✅

mamarley commented 2 years ago

Awesome! If combining the controllers is acceptable to you, I will open a PR.

nebulous commented 2 years ago

yeah - that's fine. I wrote this in Angular a zillion years ago just to try it out and have since learned that I dislike Angular :) so any patterns may be considered antipatterns.

mamarley commented 2 years ago

Ha, I don't like Angular.js (or JS in general) either… It pays the bills though.

nebulous commented 2 years ago

The latest image on dockerhub contains @mamarley's fix and works with my serial bridge. @andyopengit or @scyto - hopefully you can pull and confirm the same.

andyopengit commented 2 years ago

I have pulled the latest and can confirm that the serial information in now showing in the web interface as expected! Thanks so much for all the help and jumping on a fix so quickly @nebulous and @mamarley !

I will defer closing this since the bug label was added but can confirm that my issue was indeed resolved.

scyto commented 2 years ago

yes fixed for me, i still continue to have wierd loading delays / rendering delays but thats not related to the serial which is now flowing nicely, thanks guys.