Improve Docker compose's behaviour when running tests
Update the Spark and spark-xml version dependencies
The second is fairly straightforward, but to explain the changes in the first:
The volume definition is removed. Because the volume is persistent, it will overrule the COPY command in the Dockerfile which copies to the same location. Under the old version this meant you had to remove the volume before rebuilding if the local code had been changed. There is no reason to maintain the volume contents between runs of the container, so the volume is not necessary.
Additionally, the volume was originally mounted into the repository directory which meant that any test output, such as __pycache__ directories, was placed in the user's directory as the root user. This avoids that inconvenience.
The download_spark.sh script now sets -x, so that its commands are output before they are executed. This gives a similar output to what Docker does when running the Dockerfile.
wget is called with the "not-verbose" flag instead of the "quiet" flag. Quiet suppresses all output including errors, which in my case meant I couldn't see that I was getting a 404 for the old URL.
Hi,
This PR does two things:
The second is fairly straightforward, but to explain the changes in the first:
COPY
command in the Dockerfile which copies to the same location. Under the old version this meant you had to remove the volume before rebuilding if the local code had been changed. There is no reason to maintain the volume contents between runs of the container, so the volume is not necessary.__pycache__
directories, was placed in the user's directory as the root user. This avoids that inconvenience.download_spark.sh
script now sets-x
, so that its commands are output before they are executed. This gives a similar output to what Docker does when running the Dockerfile.wget
is called with the "not-verbose" flag instead of the "quiet" flag. Quiet suppresses all output including errors, which in my case meant I couldn't see that I was getting a 404 for the old URL.