Danger of creating embedded tomcat work directory in temporary directory on CentOS 6+

[teggr]

When using the embedded tomcat server, by default, the class TomcatEmbeddedServletContainerFactory creates a temporary directory in the "java.io.tmpdir" directory, which would likely resolve to /tmp on a linux installation. The tomcat server then places compiled jsps and the lib jars into this work directory.

I believe it is worth documenting that by default centos will clear out the /tmp directory periodically. This means that .jar and compiled jsps/tags are deleted by the tmpwatcher service. This can result in classnotfoundexceptions and other errors at run time. Most likely after 10days, which is the out of the box period.

[dsyer]

Wow, that's pretty odious. Can you paste a link to CentOS docs that say when and why this happens?

[wilkinsona]

clear out the /tmp directory periodically

AFAIK, "clear out" isn't strictly accurate. Files are only deleted if they haven't been accessed for a (configurable) period of time. @teggr I'm interested to know if you've seen this cause problems in reality or if it's a purely theoretical concern.

[teggr]

@dsyer I can't find a definitive list of out of the box packages for CentOS. Closest I can find in the list of packages and other supporting blog posts.

http://mirror.centos.org/centos/6/os/x86_64/Packages/ http://prefetch.net/blog/index.php/2009/05/01/automating-temporary-file-cleanup-with-tmpwatch/

The default CentOS minimal installation which we use on our virtual machines includes the default configuration. I believe that this is also true for Ubuntu.

#! /bin/sh
flags=-umc
/usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
        -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
        -X '/tmp/hsperfdata_*' 10d /tmp
/usr/sbin/tmpwatch "$flags" 30d /var/tmp
for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
    if [ -d "$d" ]; then
        /usr/sbin/tmpwatch "$flags" -f 30d "$d"
    fi
done

@wilkinsona is correct, it's not strictly a clear out. By default the tmpwatch service deletes files whose access time is more than 10 days old.

This is a real problem for us rather than being theoretical.

The application is packaged as an executable WAR, as we use jsps and tag libraries.

When using the jsps and tags, the compiled classes are output to the tomcat working directory.

.
└── work
    └── Tomcat
        └── localhost
            └── portal
                ├── org
                │   └── apache
                │       └── jsp
                │           ├── tag
                │           │   └── web
                │           │       ├── bootstrapform
                │           │       │   ├── checkboxgroup_tag.class
                │           │       │   ├── checkboxgroup_tag.java
                │           │       │   ├── checkbox_tag.class

After 10 days, the .java and .class file are removed if they haven't been accessed. This may be the case as Tomcat will keep a number of the compiled class files in memory. After a period of time Tomcat starts to unload or expire classes from its cache, as a result it then tries to reload the class files from the filesystem, at which point we get a ClassNotFoundException because the class files have been removed by the tmpwatch service

java.lang.ClassNotFoundException: org.apache.jsp.tag.web.widget.bpmn.variables.date_tag
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_65]
        at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:132) ~[tomcat-embed-jasper-7.0.67.jar!/:7.0.67]
        at org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:63) ~[tomcat-embed-jasper-7.0.67.jar!/:7.0.67]
        at org.apache.jsp.tag.web.widget.bpmn.variables.referral.list_tag._jspx_meth_variables_005fdate_005f0(list_tag.java:767) ~[na:na]
        at org.apache.jsp.tag.web.widget.bpmn.variables.referral.list_tag._jspx_meth_c_005fwhen_005f5(list_tag.java:742) ~[na:na]

As a work around, we are starting the application with a different temporary directory.

-Djava.io.tmpdir=/var/tmp

This allows us to continue using the default spring boot behaviour and also avoid any other possible issues caused by the tmpwatch service removing files in the tomcat working directory.

We could also remove stop or modify the tmpwatch service but we'd rather keep the default CentOS configuration at this point.

We could also use the '''server.tomcat.basedir''' to change the working directory, but I prefer the default behaviour of creating a new woring directory on restarting the application

There are these workarounds so it's potentially not an issue as such, more a documentation note perhaps?

[oatesp]

We've just encountered this issue with our application.

This occurred on an Amazon Linux instance. The application accepts multipart file uploads, requests coming through infrequently.

We encountered the error

The temporary upload location [/tmp/tomcat
.8614765871571973022.8080/work/Tomcat/localhost/ROOT] is not valid
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982)

Having found the /tmp/tomcat directory to be missing we tracked down the cause to the tmpwatch process.

For us the workaround to set a different tmp directory using -Djava.io.tmpdir=/var/tmp looks like it will help.

A difficult issue though which can come out of the blue, while we may question why our application is running for such periods without use it would be good to have some more obvious documentation to warn of this possibility.

I'm not sure if there is anything that Spring Boot could do to mitigate this issue other than warn of the possibility.

[wimdeblauwe]

Just encountered the same thing today:

Could not parse multipart servlet request; nested exception is java.io.IOException: The temporary upload location [/tmp/tomcat.3709455009677465432.8085/work/Tomcat/localhost/ROOT] is not valid

This is on a RedHat linux:

Red Hat Enterprise Linux Server release 5.9 (Tikanga)

I will try overwriting the java.io.tmpdir property to work around it.

[junkri]

I observed the same issue with Centos 6.7 as well. Please note that changing the tmp directory to /var/tmp does not help too much, as it is cleared out every 30 days: /usr/sbin/tmpwatch "$flags" 30d /var/tmp

[sirianni]

Encountered the same issue using RHEL 6.7 with file uploads failing:

2017-01-05 15:43:33.676 ERROR 23427 --- [http-nio-8081-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.multipart.MultipartException: Could not parse multipart servlet request; nested exception is java.io.IOException: The temporary upload location [/tmp/tomcat.1634813719885585463.8081/work/Tomcat/localhost/ROOT] is not valid] with root cause
      java.io.IOException: The temporary upload location [/tmp/tomcat.1634813719885585463.8081/work/Tomcat/localhost/ROOT] is not valid

I was able to workaround this by modifying /etc/cron.daily/tmpwatch with the following:

-X '/tmp/tomcat.*'

[wildcart]

sorry for posting on this closed issue:

This issue is not specific to any UNIX/LINUX distribution it is specific to the servers configuration the application (container) is running on as both the POSIX standard as well the the FHS [1] standard state

Filesystem Hierarchy Standard: 3.17 "Although data stored in /tmp may be deleted in a site-specific manner, it is recommended that files and directories located in /tmp be deleted whenever the system is booted."

I've seen a lot of servers that aren't using CentOS that specifically configure this, either by writing a cron job manually or by installing some package to do so. Complaining that it is 'plain odious' to automatically clean out temporary directories is ridiculous as it is to be expected that these files are deleted regularly, especially on larger servers which is compliant to the FHS.

To me it looks like that these files are cache files and a better place to create them on a UNIX/LINUX system seems to be /var/cache which according to FHS 5.5 "may be expired in an application specific manner." FHS 5.5 continues to argue that an application must be able to recover in case these files are deleted by the system administrator, ie "because of disk space shortage".

I think that the solution should not be to add a 'WARNING' to the documentation but to choose a better location to create these files in the first place. If this location cannot be determined automatically then it should be left unset and the application (container) should complain at start that the cache location needs to be set-up correctly.

[1] Filesystem Hierarchy Standard, especially topics 3.17, 5.15, and 5.5

[philwebb]

@wildcart thanks for the suggestions. I've opened #8459 to consider this again. I'm not keen on leaving it unset (that seems to go against Boot's "convention over configuration" philosophy) but if we can find a way to use /var/cache that might be nice.

If you want to add any additional thoughts, please do so on #8459

[ebcodes]

Resurrecting a closed issue, but for anyone who might find it useful, I've used the following with great success (spring boot / elastic beanstalk).

It will exclude the /tmp/tomcat* directories for auto-cleanup.

# Prevent the tmpwatch cron job from deleting tomcat tmp files as this causes "Internal Server Errors"
# This script overwrites the existing file (which is backed up in the same directory as tmpwatch.bak)
# 
# Place this in the root of your project under .ebextensions/tmpwatch_cron.config.
# 
# If using elastic beanstalk and spring boot:
# 1) add this to your .ebextensions/ folder with the provided name
# 2) ensure the .ebextensions folder is added as part of the zip that gets uploaded to elastic beanstalk 
#    (hint: maven-assembly-plugin to bundle as zip)
#
# see: https://github.com/spring-projects/spring-boot/issues/5009
files:
  "/etc/cron.daily/tmpwatch" :
    mode: "000755"
    owner: root
    group: root
    content: |
      #! /bin/sh
      flags=-umc
      /usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
        -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix \
        -X '/tmp/hsperfdata_*' -X '/tmp/tomcat*' 10d /tmp
      /usr/sbin/tmpwatch "$flags" 30d /var/tmp
      for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
        if [ -d "$d" ]; then
          /usr/sbin/tmpwatch "$flags" -f 30d "$d"
        fi
      done

commands:
  01_tmpwatch_command:
    command: "touch /tmp/$(date '+%F.%T.%N').tmpwatch_cmd_01"

[adamcbuckley]

If, like me, you arrived at this page because you're seeing this error while using Spring Boot, note that you can solve this issue by adding the following line to application.properties. Obviously you will substitute your own preferred tmp folder.

spring.servlet.multipart.location=/home/ec2-user/boot/tmp

More info in the Spring 2.0 documentation

[wimdeblauwe]

I tried the .ebextensions script from ebrandell, but I have still seen the error after a while. I am now using another folder instead.

• Property for Spring Boot 1.x: spring.http.multipart.location
• Property for Spring Boot 2.x: spring.servlet.multipart.location

I am using /var/app/current/tmp-tomcat as the folder. You can have Elastic Beanstalk automatically create this directory by creating the following files:

.ebextensions/10tmpfolder-tomcat.config

container_commands:
  01chmod:
    command: "chmod +x .ebextensions/tomcat/hooks/*"
  02mkdir_appdeploy_post:
    test: '[ ! -d /opt/elasticbeanstalk/hooks/appdeploy/post ]'
    command: "mkdir /opt/elasticbeanstalk/hooks/appdeploy/post"
  10appdeploy_post_createfolder:
    command: "cp .ebextensions/tomcat/hooks/10create_tmp_dir.sh /opt/elasticbeanstalk/hooks/appdeploy/post/"

_.ebextensions/tomcat/hooks/10create_tmpdir.sh

#!/usr/bin/env bash
mkdir -p /var/app/current/tmp-tomcat
chmod a+w /var/app/current/tmp-tomcat

[gillius]

As of Spring Boot 2.0.3 and 2.0.5, spring.servlet.multipart.location does not fix this issue. For whatever reason, even if giving an absolute path, Tomcat creates the location as a relative path under the same temp folder, which fails in the same way.

Instead, I had to set server.tomcat.basedir to a safe location.

This would be a lot better if Tomcat would just create the missing folders...

[wilkinsona]

@gillius Could you please open a separate issue for that with a small sample (something we can unzip or git clone) that reproduces the behaviour? Note that spring.servlet.multipart.location will only take effect if you haven't provided your own MultipartConfigElement or CommonsMultipartResolver beans.

[gillius]

I could work on that, but is it actually intended to work? I noticed in the commit 0067082eacaa1fd0650d0ee976ebabaf72d083e1, which is where I got the idea to do differently, the documentation says to use server.tomcat.basedir.

[wilkinsona]

Yes, I believe it should work. The recommendation to configure Tomcat's basedir is because it has a broader effect. It willl change the location to which compiled JSPs are written as well for example.

[gillius]

I agree, the same issue will break Tomcat for any other temp files it needs, so when I think about it, changing basedir is the only proper solution. Since that's working for me and I don't feel multipart.location is a proper solution, I won't spend the time to make a issue and sample project for it.

[ccschneidr]

Imho, this issue is not really fixed. Not cleaning temporary files is not a realistic option, because now and then (crashes, kills, bad applications) file are not cleaned by the spring boot application. The directories and files (e.g. precompiled stuff) should be recreated on demand. Generally an application cannot rely on temp files to stay there indefinitely. Is this a tomcat or a spring issue or both?

[wilkinsona]

Please take a moment to read the full history of the issue and the linked issues. For example, https://github.com/spring-projects/spring-boot/issues/9616 was opened as a result of this issue and is still open.

[nanospeck]

Just encountered the same thing today:

Could not parse multipart servlet request; nested exception is java.io.IOException: The temporary upload location [/tmp/tomcat.3709455009677465432.8085/work/Tomcat/localhost/ROOT] is not valid

This is on a RedHat linux:

Red Hat Enterprise Linux Server release 5.9 (Tikanga)

I will try overwriting the java.io.tmpdir property to work around it.

Did it work? @wimdeblauwe

[wimdeblauwe]

Yes, I am still using that which I have posted above.