tomcat has many open filehandles on known providers file

[alaniwi]

lsof shows that the tomcat java process on data node is not closing filehandles on /esg/config/esgf_known_providers.xml

Example:

• count open filehandles while avoiding double-counting with multi-threading

  # lsof -P -n -p `pidof java` | cut -b 11-15,22- | sort -u | grep -c esgf_known_providers.xml
  23

• also set strace going

  # strace -f -p `pidof java` -e '!futex' -o strace.out

• In browser, load https://<data_node>/esg-orp/home.htm

• In terminal, interrupt (ctrl-C) the strace

• count the open filehandles again -- one more than previously:

  # lsof -P -n -p `pidof java` | cut -b 11-15,22- | sort -u | grep -c esgf_known_providers.xml
  24

• see what the strace found

  # grep known_providers strace.out 
  2590  stat("/esg/config/esgf_known_providers.xml", {st_mode=S_IFREG|0644, st_size=2396, ...}) = 0
  2590  open("/esg/config/esgf_known_providers.xml", O_RDONLY) = 109
  2590  stat("/esg/config/esgf_known_providers.xml", {st_mode=S_IFREG|0644, st_size=2396, ...}) = 0
  2590  stat("/esg/config/esgf_known_providers.xml", {st_mode=S_IFREG|0644, st_size=2396, ...}) = 0
  2590  stat("/esg/config/esgf_known_providers.xml", {st_mode=S_IFREG|0644, st_size=2396, ...}) = 0

  # grep -w 109 strace.out
  [... snip ...]
  2590  open("/esg/config/esgf_known_providers.xml", O_RDONLY) = 109
  2590  fstat(109, {st_mode=S_IFREG|0644, st_size=2396, ...}) = 0
  2590  read(109, "<?xml version=\"1.0\" encoding=\"UT"..., 8192) = 2396
  2590  fstat(109, {st_mode=S_IFREG|0644, st_size=2396, ...}) = 0
  2590  lseek(109, 0, SEEK_CUR)           = 2396
  2590  fstat(109, {st_mode=S_IFREG|0644, st_size=2396, ...}) = 0
  2590  lseek(109, 0, SEEK_CUR)           = 2396
  2590  read(109, "", 8192)               = 0

  with no close

[alaniwi]

Here is where the file is used:

https://github.com/ESGF/esg-orp/blob/92c8c57dd1c8118321c376934a4412111fe7893a/web/WEB-INF/views/jsp/home.jsp#L315-L324