error with oemgateway

[modenet]

what about frequency? default value is 4... is for 433? for 866 I have to use value 2?

[lafrech]

Hi.

Thank you for your feedback.

It seems you edited your message.

Your original message contained the following error log:

root@raspbmc:~/oem_gateway-master# ./oemgateway.py 
2013-08-21 17:20:31,969 INFO Logging level set to DEBUG
2013-08-21 17:20:31,972 INFO Opening gateway...
2013-08-21 17:20:31,976 INFO Creating buffer emoncms_local
2013-08-21 17:20:31,979 INFO Creating buffer emoncms_remote
2013-08-21 17:20:31,983 INFO Creating listener RFM2Pi
2013-08-21 17:20:31,987 DEBUG Opening serial port: /dev/ttyAMA0
2013-08-21 17:20:31,992 INFO Setting send time interval to 0
2013-08-21 17:20:31,995 INFO Setting RFM2Pi | frequency: 2
2013-08-21 17:20:32,999 INFO Setting RFM2Pi | sgroup: 210
2013-08-21 17:20:34,003 INFO Setting RFM2Pi | baseid: 15
2013-08-21 17:20:35,007 INFO Creating listener Socket
2013-08-21 17:20:35,011 DEBUG Opening socket on port 50011
Traceback (most recent call last):
  File "./oemgateway.py", line 239, in <module>
    gateway.run()
  File "./oemgateway.py", line 81, in run
    values = l.read()
  File "/root/oem_gateway-master/oemgatewaylistener.py", line 137, in read
    self._rx_buf = self._rx_buf + self._ser.readline()
  File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 456, in read
    raise SerialException('device reports readiness to read but returned no data (device disconnected?)')
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected?)

And you said

I using an raspbmc distro; I only install python-serial python-configobj

I just got back home from holidays and my Pi apppears to be out of order (thunder, I'm afraid), so I can't do much about that right now. It looks as if the serial port could be opened but can't be read. I shall investigate this error to catch it properly. Does this still occur ?

I got no idea wether or not the serial port could be named differently on your distribution. Do you think /dev/ttyAMA0 is the correct path ?

Regarding your question about the frequency, you need to enter 8 for 866. These are the values used by Martin's RFM2Pi. I should explain this in the documentation.

See here:

set MHz band (4 = 433, 8 = 868, 9 = 915)

[modenet]

yes I resolved the first question: I was a hard problem with getty configuration in my raspbmc distro; I had to change /etc/init/ttyAMA0.conf to disable the use of AMA0.

thanks for the response for the second question, maybe you can add a line of comment in the conf files.

thanks again for your precisious work.

2013/8/26 Jérôme Lafréchoux notifications@github.com

Hi.

Thank you for your feedback.

It seems you edited your message. Your original message contained the following error log:

root@raspbmc:~/oem_gateway-master# ./oemgateway.py 2013-08-21 17:20:31,969 INFO Logging level set to DEBUG2013-08-21 17:20:31,972 INFO Opening gateway...2013-08-21 17:20:31,976 INFO Creating buffer emoncms_local2013-08-21 17:20:31,979 INFO Creating buffer emoncms_remote2013-08-21 17:20:31,983 INFO Creating listener RFM2Pi2013-08-21 17:20:31,987 DEBUG Opening serial port: /dev/ttyAMA02013-08-21 17:20:31,992 INFO Setting send time interval to 02013-08-21 17:20:31,995 INFO Setting RFM2Pi | frequency: 22013-08-21 17:20:32,999 INFO Setting RFM2Pi | sgroup: 2102013-08-21 17:20:34,003 INFO Setting RFM2Pi | baseid: 152013-08-21 17:20:35,007 INFO Creating listener Socket2013-08-21 17:20:35,011 DEBUG Opening socket on port 50011Traceback (most recent call last): File "./oemgateway.py", line 239, in gateway.run() File "./oemgateway.py", line 81, in run values = l.read() File "/root/oem_gateway-master/oemgatewaylistener.py", line 137, in read self._rx_buf = self._rx_buf + self._ser.readline() File "/usr/lib/python2.7/dist-packages/serial/serialposix.py", line 456, in read raise SerialException('device reports readiness to read but returned no data (device disconnected?)')serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected?)

I just got back home from holidays and my Pi apppears to be out of order (thunder, I'm afraid), so I can't do much about that right now. It looks as if the serial port could be opened but can't be read. I shall investigate this error to catch it properly. Does this still occur ?

I got no idea wether or not the serial port could be named differently on your distribution. Do you think /dev/ttyAMA0 is the correct path ?

Regarding your question about the frequency, you need to enter 8 for 866. These are the values used by Martin's RFM2Pi. I should explain this in the documentation.

See herehttps://github.com/mharizanov/RFM2Pi/blob/master/firmware/RF12_Demo_atmega328/RF12_Demo_atmega328.ino:

set MHz band (4 = 433, 8 = 868, 9 = 915)

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23248472 .

[modenet]

I have another question: some days ago I observed (when emoncms.org was down) that oemgateway stopped sending data even to my local server; I had to modify conf to stop upload to emoncms.org to resume pushing data to my local server.

[lafrech]

This is interesting.

My Pi is still unusable, so I can't investigate. This should not happen. The buffer that sends the data locally should behave normally, and the one that sends the data to the remote server should store the data (actually only a limited amount of data) until the remote site is up again, then send it.

It could be that the timeout is so long before giving up remote sending that there is no time left between to sets of data for a local sending. What is the frequency of your sampling (how many seconds between to RF frames) ?

I tried to make the timeout shorter, but the timeout parameter of the urllib function just wouldn't work as I expected. I investigated quickly and found people with the same issue but no satisfying answer. This is on my TODO list (https://github.com/Jerome-github/oem_gateway/issues/1). Not sure it is the cause of the issue you're describing, though.

[modenet]

Hi, thank you for your interesting. I'm using a standard emon tx 3ct+power sketch, so 5 seconds between two RF frames).

[lafrech]

I think the timeout is 10 seconds, so this could be the cause. This timeout issue definitely needs to be fixed.

If you didn't stop the application but just removed the remote buffer from the config file, then the local data was still in RAM (in the local buffer) and should have been sent to the local emoncms all at once after you removed the remote buffer. If this did not happen, then it is an issue.

Not sure I'm clear, but the "buffer" is designed to keep the data when the application can't send it. This happens if the network is down, but it should work just as well if the cause for the data not being sent is that the gateway doesn't have the time to send it (due to the timeout being too long).

Can you tell if all the data was lost in the period when emoncms.org was down ?

[modenet]

on emoncms.prg down I saw that oemgateway stop also sending data to my local emoncms. after I disabled emoncms.org transmission and restarted oemgateway, my local server resumed receiving data.

yes, I lost data in local emoncms and in the remote one during emoncms.orgdown (it remained down for a day).

2013/8/29 Jérôme Lafréchoux notifications@github.com

I think the timeout is 10 seconds, so this could be the cause. This timeout issue definitely needs to be fixed.

If you didn't stop the application but just removed the remote buffer from the config file, then the local data was still in RAM (in the local buffer) and should have been sent to the local emoncms all at once after you removed the remote buffer. If this did not happen, then it is an issue.

Not sure I'm clear, but the "buffer" is designed to keep the data when the application can't send it. This happens if the network is down, but it should work just as well if the cause for the data not being sent is that the gateway doesn't have the time to send it (due to the timeout being too long).

Can you tell if all the data was lost in the period when emoncms.org was down ?

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23490719 .

[lafrech]

Actually, you didn't have to restart the gateway (and lose the data). You could modify the config file and wait for the gateway to reload it (every second, a bit more if stuck in urllib timeout). Then the remote buffer would have been deleted and the local buffer would have went on working normally.

I admit this is not obvious. Generally, a config file is just read on startup.

[modenet]

WOW! have you released a new version with these features or them were already present (and I did not knows)?

2013/8/29 Jérôme Lafréchoux notifications@github.com

Actually, you didn't have to restart the gateway (and lose the data). You could modify the config file and wait for the gateway to reload it (every second, a bit more if stuck in urllib timeout). Then the remote buffer would have been deleted and the local buffer would have went on working normally.

I admit this is not obvious. Generally, a config file is just read on startup.

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23496937 .

[lafrech]

This is in the version you are using. I didn't do any work recently.

The parameters are checked continuously. This behaviour is inherited from the original version, where the parameters can be changed at any time through the emoncms GUI.

The data being kept in the buffer when it can not be send is a somehow half-done feature. Right now, it is kept in RAM until it gets too big (I hardcoded an arbitrary limit). It would be nice to allow the buffer to write data in a file in this case, instead of trashing it. This should be on the TODO list.

[modenet]

ok, my mistake was to stop oemgateway after changing configuration file. your work is fastanstic!

2013/8/29 Jérôme Lafréchoux notifications@github.com

This is in the version you are using. I didn't do any work recently.

The parameters are checked continuously. This behaviour is inherited from the original version, where the parameters can be changed at any time through the emoncms GUI.

The data being kept in the buffer when it can not be send is a somehow half-done feature. Right now, it is kept in RAM until it gets too big (I hardcoded an arbitrary limit). It would be nice to allow the buffer to write data in a file in this case, instead of trashing it. This should be on the TODO list.

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23499799 .

[modenet]

I have another question, I try to explain you: today I activated an EMONGlcd with standard PV sketch; after I activated on oemgateway time send every 30 sec. on emonglcd I saw that time was not recognised so I decided to restart oemgateway. now time is recognise in strange mode: hours are always 0, minutes are corrects.

what happens? in other emonglcd with same skecth I have no problem to receive time from a raspberry with emoncms pre-builed image.

thank you again

2013/8/29 Denis Moro denis.moro@gmail.com

ok, my mistake was to stop oemgateway after changing configuration file. your work is fastanstic!

2013/8/29 Jérôme Lafréchoux notifications@github.com

This is in the version you are using. I didn't do any work recently.

The parameters are checked continuously. This behaviour is inherited from the original version, where the parameters can be changed at any time through the emoncms GUI.

The data being kept in the buffer when it can not be send is a somehow half-done feature. Right now, it is kept in RAM until it gets too big (I hardcoded an arbitrary limit). It would be nice to allow the buffer to write data in a file in this case, instead of trashing it. This should be on the TODO list.

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23499799 .

[lafrech]

Can you please have a look in the log to see what exactly is sent by the gateway ?

Search for

Broadcasting time:

If the time here is correct, maybe you'll want to print what is actually written in the serial port:

https://github.com/Jerome-github/oem_gateway/blob/master/oemgatewaylistener.py#L285

After the following line,

self._ser.write("%02d,00,%02d,00,s" % (now.hour, now.minute))

add

print("%02d,00,%02d,00,s" % (now.hour, now.minute))

for a display in the terminal (if you're running the gateway from the terminal) or

self._log.debug("%02d,00,%02d,00,s" % (now.hour, now.minute))

to have it printed in the logs. Both not tested, but should work.

The result should look as described here:

http://harizanov.com/2013/02/new-rfm2pi-board-in-the-works/#comment-669

[modenet]

yes, both are corrects:

my local time is 14:25 2013-09-02 14:25:26,467 DEBUG Broadcasting time: 14:25 2013-09-02 14:25:26,469 DEBUG 14,00,25,00,s

how do you explain what happens to my emonglcd?

2013/9/2 Jérôme Lafréchoux notifications@github.com

Can you please have a look in the log to see what exactly is sent by the gateway ?

Search for

Broadcasting time:

If the time here is correct, maybe you'll want to print what is actually written in the serial port:

https://github.com/Jerome-github/oem_gateway/blob/master/oemgatewaylistener.py#L285

After the following line,

self._ser.write("%02d,00,%02d,00,s" % (now.hour, now.minute))

add

print("%02d,00,%02d,00,s" % (now.hour, now.minute))

for a display in the terminal (if you're running the gateway from the terminal) or

self._log.debug("%02d,00,%02d,00,s" % (now.hour, now.minute))

to have it printed in the logs. Both not tested, but should work.

The result should look as described here:

http://harizanov.com/2013/02/new-rfm2pi-board-in-the-works/#comment-669

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23656089 .

[lafrech]

I have no idea. I don't own any emonGLCD myself.

You may want to ask on the OEM forum, with an excerpt of the log.

Perhaps can you try to instrument the code (add print statements to watch variables through the serial link) in the emonGLCD to see what exactly comes in and spot a potential flaw in the processing, but I doubt there would be an error there.

what happens? in other emonglcd with same skecth I have no problem to receive time from a raspberry with emoncms pre-builed image.

Did you try the other emonGLCDs with this software, or this GLCD with the ready-to-go image ?

[modenet]

Hi, I did few debug to emonglcd pv sketch adding these lines:

  if (node_id == 15)
  {
    RTC.adjust(DateTime(2013, 1, 1, rf12_data[1], rf12_data[2],

rf12_data[3])); last_emonbase = millis(); Serial.println("time received"); Serial.println(rf12_data[1]); Serial.println(rf12_data[2]); Serial.println(rf12_data[3]);

and the output was always:

time received 0 23 0

so the problem was in the data sended. so I modified line 285 of oemgatewaylistener.py from self._ser.write("%02d,00,%02d,00,s" % (now.hour, now.minute)) to self._ser.write("00,%02d,%02d,00,s" % (now.hour, now.minute))

and now my emonglcd works fine!!!!

the serial output is correct: time received 22 23 0

2013/9/2 Jérôme Lafréchoux notifications@github.com

I have no idea. I don't own any emonGLCD myself.

You may want to ask on the OEM forum, with an excerpt of the log.

Perhaps can you try to instrument the code (add print statements to watch variables through the serial link) in the emonGLCD to see what exactly comes in and spot a potential flaw in the processing, but I doubt there would be an error there.

what happens? in other emonglcd with same skecth I have no problem to receive time from a raspberry with emoncms pre-builed image.

Did you try the other emonGLCDs with this software, or this GLCD with the ready-to-go image ?

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23658353 .

[lafrech]

Well spotted !

And the php implementation seems to confirm your findings.

Here is where I got the format from.

I shall commit your fix soon. Just need to get my Pi up running. And have the fix confirmed by Trystan or Martin.

Thanks for investigating this and sorry for the inconvenience.

[modenet]

thank you for your works!

2013/9/3 Jérôme Lafréchoux notifications@github.com

Well spotted !

And the php implementationhttps://github.com/emoncms/raspberrypi/blob/master/raspberrypi_run.php#L313seems to confirm your findings.

Herehttp://harizanov.com/2013/02/new-rfm2pi-board-in-the-works/#comment-658is where I got the format from.

I shall commit your fix soon. Just need to get my Pi up running. And have the fix confirmed by Trystan or Martin.

Thanks for investigating this and sorry for the inconvenience.

— Reply to this email directly or view it on GitHubhttps://github.com/Jerome-github/oem_gateway/issues/3#issuecomment-23695078 .

[lafrech]

Should be solved, now. Thanks for sharing the fix.

https://github.com/Jerome-github/oem_gateway/commit/43eb3c6d51242c9c6795f4cfca35dbc189db8292

Obviously, I couldn't test, as I don't own an emonGLCD myself.