search

dsa110 / dsa110-issues

Issue tracker for all DSA-110 work
0 stars 0 forks source link

bokehservice server out of disk space #81

Open caseyjlaw opened 3 years ago

caseyjlaw commented 3 years ago

I seem to recall this error has come up on other containers...

May 11 21:47:21 bokehservice syslog-ng[1334]: I/O error occurred while writing; fd='16', error='No space left on device (28)'
May 11 21:47:21 bokehservice syslog-ng[1334]: Suspending write operation because of an I/O error; fd='16', time_reopen='60'

This isn't breaking the bokeh service, but good if we can apply the fix to this one, too.

rh-codebase commented 3 years ago

the bokeh service is logging heavily with: Mar 26 22:10:32 bokehservice bokeh[1174]: 2021-03-26T22:10:32 [error ] {"mjd": 59299.92399213147, "proj": "dsa", "subsystem": "System", "app": "-", "version": "-", "module": "dsaStore", "function": "get_dict", "msg": "could not convert json to dictionary"} Mar 26 22:10:32 bokehservice bokeh[1174]: 2021-03-26T22:10:32 [warning ] {"mjd": 59299.9239921469, "proj": "dsa", "subsystem": "software", "app": "-", "version": "-", "module": "vis", "function": "-", "msg": "get_dict returned nonstandard ant dict"} More debugging needed as to why the antenna monitor stream is no longer a standard dictionary.

caseyjlaw commented 3 years ago

Related to issue #19