Closed rvizarreta closed 4 months ago
Notes:
This is not a code issue, this is due to the limit in files.
This is done, by solving this issue I essentially encapsulated all possible failures and, most important, I added this execute command function:
def execute_command(self, command):
try:
output = check_output(command, shell=True, text=True)
return output
except subprocess.CalledProcessError as e:
print(f"Error executing command: {e}")
output = self.execute_command(command)
return output
This way if we get an error on making the snmp request at least we will try again until connection is succesful. I also added time delays so request are done more carefully now. This doesn't entirely fix the issue and affects our sampling rate drastically, but it does the job and is good enough for low voltage monitoring.
Some learnings:
lsof | awk '{ print $1 " " $2; }' | sort -rn | uniq -c | sort -rn | head -15
you can see how many processes and files are open, this helped me noticing that python was taking a considerable amount of files open, where each snmp command is treated as 'opening a file' on Linux.ulimit -n
, ulimit -Sn
, ulimit -Hn
, to see and change the number of files allowed by user, but this was not helpful.Ref: https://www.howtogeek.com/805629/too-many-open-files-linux/
Remote controls still need testing.
From local work: JSON file is not ready for this one. Also, remember from Louise: "in the final set up we will have 3 VME crates (one for ADCs, two for the VGAs) so we need a few more boxes for those." Manual: https://file.wiener-d.com/documentation/Crates_6000/WIENER_Series_6000_Manual_A7.pdf