Suggested changes for production

[ccare]

Hi @ceddlyburge,

I've been trying to get the test-runner running in production, and have hit one or two issues. I guess you'll know more about how we might get around them.

The first issue is that the the Elm runtime seems to need to write to $HOME/.elm, so we'll need to override this location there maybe an Elm specific way of doing this, but as a quick workaround, I found that setting the home folder to /mnt/exercism-iteration works for now.

e.g.

on https://github.com/exercism/elm-test-runner/blob/master/run.sh#L9 insert

export HOME=/mnt/exercism-iteration

The second is that the Elm runtime appears to want to download packages from the internet. As the runtime container has no network access, this is a little problematic. Do the required packages change on an exercise-by-exercise basis, or could they be pre-installed?

You can reach me on slack if you want to work through this 1-1. I can try things out directly in an environment that closely matches the runtime environment.

Thanks,

Charles

[mpizenberg]

Setting the HOME env var should work indeed. Regarding the binary, if there is no internet access and the libraries are already downloaded in that $HOME/.elm it should work. Currently, I believe the dependencies listed by this elm.json are sufficient for all exercism exercises, including the research experiment.

{
    "type": "application",
    "source-directories": [
        "src"
    ],
    "elm-version": "0.19.1",
    "dependencies": {
        "direct": {
            "elm/browser": "1.0.2",
            "elm/core": "1.0.4",
            "elm/html": "1.0.0",
            "elm/parser": "1.1.0",
            "elm/regex": "1.0.0",
            "elm/time": "1.0.0"
        },
        "indirect": {
            "elm/json": "1.1.3",
            "elm/url": "1.0.0",
            "elm/virtual-dom": "1.0.2"
        }
    },
    "test-dependencies": {
        "direct": {
            "elm-explorations/test": "1.2.2",
            "rtfeldman/elm-iso8601-date-strings": "1.1.3"
        },
        "indirect": {
            "elm/random": "1.0.0"
        }
    }
}

The following archive is a 500Kb file containing a .elm/ dir with exactly the above dependencies. If this is decompressed in the home directory, I believe everything should work. elm.tar.gz

[ceddlyburge]

Thanks matthieu, I'll get on to that.

I'm kind of surprised it isn't working without internet, being as the docker image runs on my machine, and there won't be any internet access inside the container.

But there you go, it sounds like the fix is quite simple anyway.

Cheers, Cedd

On Wed, 15 Jan 2020, 19:07 Matthieu Pizenberg, notifications@github.com wrote:

Setting the HOME env var should work indeed. Regarding the binary, if there is no internet access and the libraries are already downloaded in that $HOME/.elm it should work. Currently, I believe the dependencies listed by this elm.json are sufficient for all exercism exercises, including the research experiment.

{ "type": "application", "source-directories": [ "src" ], "elm-version": "0.19.1", "dependencies": { "direct": { "elm/browser": "1.0.2", "elm/core": "1.0.4", "elm/html": "1.0.0", "elm/parser": "1.1.0", "elm/regex": "1.0.0", "elm/time": "1.0.0" }, "indirect": { "elm/json": "1.1.3", "elm/url": "1.0.0", "elm/virtual-dom": "1.0.2" } }, "test-dependencies": { "direct": { "elm-explorations/test": "1.2.2", "rtfeldman/elm-iso8601-date-strings": "1.1.3" }, "indirect": { "elm/random": "1.0.0" } } }

The following archive is a 500Kb file containing a .elm/ dir with exactly the above dependencies. If this is decompressed in the home directory, I believe everything should work. elm.tar.gz https://github.com/exercism/elm-test-runner/files/4066901/elm.tar.gz

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=ADHR5RY5XFGS2O2KE4JJQTTQ55NHVA5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJBOKYA#issuecomment-574809440, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHR5R46TCUDOXX5XN6M5JTQ55NHVANCNFSM4KHCBC7Q .

[ceddlyburge]

Hi Charles, I pushed a commit on my way to work this morning which should address this, and the docker image builds ok. I haven't had time to test that the runner still works, but I assume it does. I'll give it another look on the train home this evening.

[iHiD]

(@ceddlyburge FYI, @ccare is off until next week now, so you probably won't get a response until then.)

[mpizenberg]

@ceddlyburge For git history purposes to have clean diffs, and for size, I'm wondering if we could add the compressed archive instead to the repo, and decompress it when building the docker image. I'm wondering if we could even not store it directly in the repo and curl it from github storage at build time since it lives at https://github.com/exercism/elm-test-runner/files/4066901/elm.tar.gz.

Both approaches would require git history rewrite to be worth the effort.

[ceddlyburge]

Hi Everyone.

I've checked and this should now work. I've run run-in-docker.ps1 on my machine with the wifi turned off, and it works as expected. run-in-docker.sh doesn't work on my windows machine, but it looks like it will work, and is not required for the test runner in any case (although it is useful to check that everything is working).

I did get the error message below when initially testing which was due to my code being written for Elm 0.19.0, while 0.19.1 is used in the test runner. Once I fixed up the code to work with 0.19.1

-- CONFUSING FILE --------------------------------------------------------------

I am getting confused when I try to compile this file:

    /solution/tests/Tests.elm

I always check if files appear in any of the "source-directories" listed in your
elm.json to see if there might be some cached information about them. That can
help me compile faster! But in this case, it looks like this file may be in
either of these directories:

    /solution
    /solution/tests

Try to make it so no source directory contains another source directory!

Compilation failed while attempting to build /solution/tests/Tests.elm

[ceddlyburge]

@ceddlyburge For git history purposes to have clean diffs, and for size, I'm wondering if we could add the compressed archive instead to the repo, and decompress it when building the docker image.

Hi Matthieu, I don't think I 100% agree with this, but I am not really opposed to it either. Can we park it until the test runner is known to be working inside the exercism ecosystem to keep things simple? Then once it is working we can iterate ....

[mpizenberg]

Hi Matthieu, I don't think I 100% agree with this, but I am not really opposed to it either. Can we park it until the test runner is known to be working inside the exercism ecosystem to keep things simple? Then once it is working we can iterate ....

I'm fine with this. Thank you for taking care of this test runner!

[iHiD]

Can we park it until the test runner is known to be working inside the exercism ecosystem to keep things simple?

Just as a note on this, once something's in git, we can't take it out. It's there for good. Some there's no coming back to it later. Other tracks haven't realised that in the past and regretted having things there, so I'm just raising this for information - I don't really have an opinion personally here.

[ceddlyburge]

Hi Jeremy, it's already in git! Do you know what the concerns usually are? Is it leaking confidential information, or appearing unprofessional or something like that?

I can't forsee any problems at the moment, but maybe if you mention something things would become clearer.

[iHiD]

The issue is that to download the repo to work on it, someone now needs to download dozens of megabytes of files. And for some people this will be slow and costly.

Whereas if the files are stored separately to the repo, then someone can work on the test-running part of the repo, with a much smaller download footprint, without worrying about the deployment dependencies.

I don't think adding the archive itself to the repo would help (in fact I think it would make it worse), but storing the deployment files seperately in s3 or similar might work well.

See https://github.com/exercism/ocaml/issues/300 for an example of where this frustrated people later down the line.

[ceddlyburge]

Yeah fair enough, I agree its generally best to keep binaries and things like that out of repos.

This repo is still small (under 1mb when I downloaded the zip), and there are useful things that could be done, like using git lfs and leveraging the clone depth.

I also think the history of this repo could be rewritten if necessary, as its the test runner and not the exercises.

And keeping things together makes things a lot simpler to maintain and develop when compared to an s3 style solution.

Personally I'm happy with it as it is, but I guess I probably have a pretty good internet connection.

I think we should leave it as it is until it is working, and then iterate from a solid foundation. The repo isn't big yet, so if we move to an s3 style solution soon only a little damage will have been done. If something bad happens I think rewriting the history would probably be a possibility, although its not ideal (and may not work at all for some exercise reason I don't know about).

Cheers, Cedd

On Fri, 17 Jan 2020, 11:06 Jeremy Walker, notifications@github.com wrote:

The issue is that to download the repo to work on it, someone now needs to download dozens of megabytes of files. And for some people this will be slow and costly.

Whereas if the files are stored separately to the repo, then someone can work on the test-running part of the repo, with a much smaller download footprint, without worrying about the deployment dependencies.

I don't think adding the archive itself to the repo would help (in fact I think it would make it worse), but storing the deployment files seperately in s3 or similar might work well.

See exercism/ocaml#300 https://github.com/exercism/ocaml/issues/300 for an example of where this frustrated people later down the line.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=ADHR5R2COGWXQIGFHKLP6ODQ6GGK5A5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJHK5CI#issuecomment-575581833, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHR5RYZNYWGAXUZ44MMGI3Q6GGK5ANCNFSM4KHCBC7Q .

[iHiD]

And keeping things together makes things a lot simpler to maintain and develop when compared to an s3 style solution.

I think we should leave it as it is until it is working, and then iterate from a solid foundation. The repo isn't big yet, so if we move to an s3 style solution soon only a little damage will have been done.

I agree with both of these points :)

If something bad happens I think rewriting the history would probably be a possibility, although its not ideal

The Test Runner versions are linked to the shas of the git commits they're built from, so it wouldn't be ideal, but I'm not sure it would be terminal.

As I said at the beginning, I genuinely don't have an opinion on this. I'll trust what y'all think is best. I just wanted to raise the fact that with it going in it can't come out, because people often don't realise that git works in that way, and the permeance of a commit.

[ceddlyburge]

Cool cool,thanks Jeremy

On Fri, 17 Jan 2020, 17:18 Jeremy Walker, notifications@github.com wrote:

And keeping things together makes things a lot simpler to maintain and develop when compared to an s3 style solution.

I think we should leave it as it is until it is working, and then iterate from a solid foundation. The repo isn't big yet, so if we move to an s3 style solution soon only a little damage will have been done.

I agree with both of these points :)

If something bad happens I think rewriting the history would probably be a possibility, although its not ideal

The Test Runner versions are linked to the shas of the git commits they're built from, so it wouldn't be ideal, but I'm not sure it would be terminal.

As I said at the beginning, I genuinely don't have an opinion on this. I'll trust what y'all think is best. I just wanted to raise the fact that with it going in it can't come out, because people often don't realise that git works in that way, and the permeance of a commit.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=ADHR5R527IGHPUDA3NCNZYDQ6HR5JA5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJIL44Y#issuecomment-575716979, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHR5R2HZVF4YDUJHC7XXYDQ6HR5JANCNFSM4KHCBC7Q .

[ccare]

Hi @ceddlyburge,

Ok - so I've been having a play with this. We're still not quite there, but close. I've dropped into an interactive shell to try and work it out.

If I set the home dir as you suggested, we run into problems with the filesystem being read-only

...
elm: /opt/test-runner/elm-home/.elm: createDirectory: permission denied (Read-only file system)
...

So I've tried using the input folder ($2) which is read-write

cd /mnt/exercism-iteration
cp -R /opt/test-runner/elm-home/ .
export HOME="/mnt/exercism-iteration/elm-home"

But then I get this

/mnt/exercism-iteration # elm-test

-- PROBLEM LOADING PACKAGE LIST ------------------------------------------------

I need the list of published packages to verify your dependencies, so I tried to
fetch:

    https://package.elm-lang.org/all-packages

But my HTTP library is giving me the following error message:

    ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "package.elm-lang.org", service name: Just "443"): does not exist (Try again)

Are you somewhere with a slow internet connection? Or no internet? Does the link
I am trying to fetch work in your browser? Maybe the site is down? Does your
internet connection have a firewall that blocks certain domains? It is usually
something like that!

Compilation failed while attempting to build /mnt/exercism-iteration/tests/Tests.elm

If I change the home folder to /mnt/exercism-iteration and manually make the .elm folder (and copy in the 0.19.1 folder), I can get this error message (which feels the closest)

/mnt/exercism-iteration # elm-test
-- TROUBLE VERIFYING DEPENDENCIES ------------------------------------- elm.json

I could not connect to https://package.elm-lang.org to get the latest list of
packages, and I was unable to verify your dependencies with the information I
have cached locally.

Are you able to connect to the internet? These dependencies may work once you
get access to the registry!

Note: If you changed your dependencies by hand, try to change them back! It is
much more reliable to add dependencies with elm install or the dependency
management tool in elm reactor.

Compilation failed while attempting to build /mnt/exercism-iteration/tests/Tests.elm

I'm struggling to know if these messages mean that a dependency is missing, or present and not verified, whatever that means in this context. Any thoughts? Is there an offline running option in the elm toolchain? I've googled around, but not found it.

Appreciate any help you can offer!

[mpizenberg]

Hi all, I'll be able to have a look at this Sunday I think

On Thu, Jan 23, 2020, 12:34 Charles Care notifications@github.com wrote:

Hi @ceddlyburge https://github.com/ceddlyburge,

Ok - so I've been having a play with this. We're still not quite there, but close. I've dropped into an interactive shell to try and work it out.

If I set the home dir as you suggested, we run into problems with the filesystem being read-only

... elm: /opt/test-runner/elm-home/.elm: createDirectory: permission denied (Read-only file system) ...

So I've tried using the input folder ($2) which is read-write

cd /mnt/exercism-iteration cp -R /opt/test-runner/elm-home/ . export HOME="/mnt/exercism-iteration/elm-home"

But then I get this

/mnt/exercism-iteration # elm-test

-- PROBLEM LOADING PACKAGE LIST ------------------------------------------------

I need the list of published packages to verify your dependencies, so I tried to fetch:

https://package.elm-lang.org/all-packages

But my HTTP library is giving me the following error message:

ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "package.elm-lang.org", service name: Just "443"): does not exist (Try again)

Are you somewhere with a slow internet connection? Or no internet? Does the link I am trying to fetch work in your browser? Maybe the site is down? Does your internet connection have a firewall that blocks certain domains? It is usually something like that!

Compilation failed while attempting to build /mnt/exercism-iteration/tests/Tests.elm

If I change the home folder to /mnt/exercism-iteration and manually make the .elm folder (and copy in the 0.19.1 folder), I can get this error message (which feels the closest)

/mnt/exercism-iteration # elm-test -- TROUBLE VERIFYING DEPENDENCIES ------------------------------------- elm.json

I could not connect to https://package.elm-lang.org to get the latest list of packages, and I was unable to verify your dependencies with the information I have cached locally.

Are you able to connect to the internet? These dependencies may work once you get access to the registry!

Note: If you changed your dependencies by hand, try to change them back! It is much more reliable to add dependencies with elm install or the dependency management tool in elm reactor.

Compilation failed while attempting to build /mnt/exercism-iteration/tests/Tests.elm

I'm struggling to know if these messages mean that a dependency is missing, or present and not verified, whatever that means in this context. Any thoughts? Is there an offline running option in the elm toolchain? I've googled around, but not found it.

Appreciate any help you can offer!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=AAWFOCKZV777MBNFRTLE5WLQ7F6DHA5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJXCILI#issuecomment-577643565, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWFOCOYOIYQPSRX5JGM7H3Q7F6DHANCNFSM4KHCBC7Q .

[ceddlyburge]

Hi Charles

How are you running this? I can run it, in Docker, on my computer, so I am confused as to why it isn't working for you (being as Docker, is, well Docker). Presumably you are creating the Docker image yourself (which is a source of difference), we probably have different Docker versions (I am on windows for example), and are maybe using different settings in the docker build command. Maybe I could upload an image to a registry somewhere (is there an exercism one we could use), then you could download it and run that. If this works it would verify that the image creation process is the problem.

There should be no need to set the HOME environment variable, as this is already present in the Docker image.

I think the file system isn't read only inside the Docker container that I create.

Thanks, Cedd

On Thu, Jan 23, 2020 at 11:34 AM Charles Care notifications@github.com wrote:

Hi @ceddlyburge https://github.com/ceddlyburge,

Ok - so I've been having a play with this. We're still not quite there, but close. I've dropped into an interactive shell to try and work it out.

If I set the home dir as you suggested, we run into problems with the filesystem being read-only

... elm: /opt/test-runner/elm-home/.elm: createDirectory: permission denied (Read-only file system) ...

So I've tried using the input folder ($2) which is read-write

cd /mnt/exercism-iteration cp -R /opt/test-runner/elm-home/ . export HOME="/mnt/exercism-iteration/elm-home"

But then I get this

/mnt/exercism-iteration # elm-test

-- PROBLEM LOADING PACKAGE LIST ------------------------------------------------

I need the list of published packages to verify your dependencies, so I tried to fetch:

https://package.elm-lang.org/all-packages

But my HTTP library is giving me the following error message:

ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "package.elm-lang.org", service name: Just "443"): does not exist (Try again)

Are you somewhere with a slow internet connection? Or no internet? Does the link I am trying to fetch work in your browser? Maybe the site is down? Does your internet connection have a firewall that blocks certain domains? It is usually something like that!

Compilation failed while attempting to build /mnt/exercism-iteration/tests/Tests.elm

If I change the home folder to /mnt/exercism-iteration and manually make the .elm folder (and copy in the 0.19.1 folder), I can get this error message (which feels the closest)

/mnt/exercism-iteration # elm-test -- TROUBLE VERIFYING DEPENDENCIES ------------------------------------- elm.json

I could not connect to https://package.elm-lang.org to get the latest list of packages, and I was unable to verify your dependencies with the information I have cached locally.

Are you able to connect to the internet? These dependencies may work once you get access to the registry!

Note: If you changed your dependencies by hand, try to change them back! It is much more reliable to add dependencies with elm install or the dependency management tool in elm reactor.

Compilation failed while attempting to build /mnt/exercism-iteration/tests/Tests.elm

I'm struggling to know if these messages mean that a dependency is missing, or present and not verified, whatever that means in this context. Any thoughts? Is there an offline running option in the elm toolchain? I've googled around, but not found it.

Appreciate any help you can offer!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=ADHR5R4XANHFPWNP57KBCCTQ7F6DHA5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJXCILI#issuecomment-577643565, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHR5R5I3MY4IBXEYMOQS6LQ7F6DHANCNFSM4KHCBC7Q .

[ccare]

Hi Cedd,

We're not running in Docker, I'm afraid. This does mean that any env vars set in the Docker chain don't make it into production. We're making an OCI image and then invoking with runc (and therefore have to set env vars).

But we're also running with an immutable disk, we can't have /opt/test-runner/ changing between runs. Can you get your version working with the dependencies copied into the $2 path (currently /mnt/exercism-iteration). If you can, that should help.

Just to clarify, what needs to be in .elm to stop the unit test runner hitting the network?

Thanks,

Charles

[ceddlyburge]

Hi Charles, this is confusing, I thought the point was to get it running in Docker!

It definitely works in docker, on my mahine, with the wifi turned off.

I think there might be a problem when the internet is available, but the site can't be reached, or there is a timeout or something. Basically, if it seems possible to connect, Elm will try and then error if it can't. If it's obvious that it can't connect and it has everything in the cache, then it doesn't worry,

Can you try turning the internet off entirely in the system you are using (I'm afraid I don't understand the terms you use)

Thanks, Cedd

[ccare]

Hi Cedd,

Apologies for being confusing. There are, in fact, only tiny differences, it's just they seem to be causing a lot of problems for you. Sorry about that.

If it runs in Docker, it will be containerised and should run in our environment (albeit with the odd tweak). We use the Dockerfile to make the image, but the image doesn't include the runtime aspects of the Dockerfile (e.g. the env vars). So for runtime env vars (like the override of $HOME) the easiest thing is to script them up. We could set them externally, but by putting them in the script we can allow the maintainers to keep control (and have a single runtime config for how we invoke all the runners).

There are 3 things causing pain, the first two are the main ones

• no network in our production environment. Clearly this is fine for you and not for me. Which makes me wonder if the structure of the .elm folder in home is slightly wrong for me.

• readonly file system for root (/) so elm cannot create lock files in the bundle of packages you've added

• the lack of env vars, this is easily fixed by adding env vars to the run.sh

If you could try invoking docker slightly differently, you should be able to mimic our production setup.

I would suggest removing your ENV entry in the Dockerfile and invoking it like this.

docker run \
    -e HOME=/opt/test-runner/elm-home \
    --network none \
    --read-only \
    --mount type=bind,src=$PWD/$2,dst=/solution \
    --mount type=bind,src=$PWD/$3,dst=/output \
    elm-test-runner $1 /solution /output

I suspect this will blow up because /opt/test-runner/elm-home is not writable and you might see some of the same errors as me.

If you could provide me with sample input (I really don't know anything about elm) for the /solution folder, I'll happily do some testing at my end too.

[ceddlyburge]

Hi Charles

I am looking in to this at the moment. The docker command you quote above works for me, without making any changes to the Dockerfile.

However I can make it fail by making the solution directory read only, like this:

docker run --network none --name=elm-test-runner --detach --read-only --mount type=bind,src=C:\Users\ceddl\Documents\GITHub\leap,dst=/solution,readonly --mount type=bind,src=C:\Users\ceddl\Documents\GITHub\elm-test-runner,dst=/output elm-test-runner leap /solution /output

This is because Elm will try and create an elm-stuff directory in the solution folder. Creating build artifacts in the same directory (or a descendant one) is a super common paradigm though, and there is nothing in the docs about the solution directory being read only, so I'm not sure if this is relevevant or not.

Please let me know if the solution folder is read only in your system

I have attached the C:\Users\ceddl\Documents\GITHub\leap directory I mention above. Maybe if you run it with some actual code it will work better. Again, please let me know!

Thanks, Cedd

leap.zip

[ceddlyburge]

Maybe you could confirm / deny that running the docker image (with your quoted parameters) works for you?

[ccare]

Sure. I'll give it a try later.

Thanks,

Charles

On Fri, 24 Jan 2020, 13:52 Cedd Burge, notifications@github.com wrote:

Maybe you could confirm / deny that running the docker image (with your quoted parameters) works for you?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=AADLWQSBZGPD6UBXJ4OEMNTQ7LXDXA5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ227KI#issuecomment-578138025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADLWQVCGM6UISJRMW4I53LQ7LXDXANCNFSM4KHCBC7Q .

[ccare]

Hi Cedd,

I checked out the code on my mac (docker 19.03.1), and built and tagged it

docker build -t elm-test-runner .

Then I created and out and leap directory for the output and input. ./leap contains your zip file.

Then I invoked it as this

docker run --mount type=bind,src=`pwd`/leap,dst=/solution \
                   --mount type=bind,src=`pwd`/out,dst=/output \
                   -e HOME=/opt/test-runner/elm-home \
                   --network none --read-only elm-test-runner leap /solution /output

And with that, I get this (which, I think, is pretty similar to what I was seeing in production):

$ docker run --mount type=bind,src=`pwd`/leap,dst=/solution --mount type=bind,src=`pwd`/out,dst=/output -e HOME=/opt/test-runner/elm-home --network none --read-only elm-test-runner leap /solution /output
leap, /solution, /output
Running tests
elm: /opt/test-runner/elm-home/.elm: createDirectory: permission denied (Read-only file system)

-- ERROR -----------------------------------------------------------------------

I ran into something that bypassed the normal error reporting process! I
extracted whatever information I could from the internal error:

>   thread blocked indefinitely in an MVar operation

These errors are usually pretty confusing, so start by asking around on one of
forums listed at https://elm-lang.org/community to see if anyone can get you
unstuck quickly.

-- REQUEST ---------------------------------------------------------------------

If you are feeling up to it, please try to get your code down to the smallest
version that still triggers this message. Ideally in a single Main.elm and
elm.json file.

From there open a NEW issue at https://github.com/elm/compiler/issues with your
reduced example pasted in directly. (Not a link to a repo or gist!) Do not worry
about if someone else saw something similar. More examples is better!

This kind of error is usually tied up in larger architectural choices that are
hard to change, so even when we have a couple good examples, it can take some
time to resolve in a solid way.elm: thread blocked indefinitely in an MVar operation
Compilation failed while attempting to build /solution/tests/Tests.elm
Converting Junit output to exercism output
Traceback (most recent call last):
  File "/opt/test-runner/bin/process_results.py", line 60, in <module>
    convert_junitxml_to_json(opts.xml_path, opts.output_path)
  File "/opt/test-runner/bin/process_results.py", line 45, in convert_junitxml_to_json
    tree = ET.parse(xml_path)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 598, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
Finished

[ccare]

^ I think if we could move the elm-home to the read-write parts of the image (using the input directory should be fine) then I think we'll be able to resolve the read-only issue.

e.g. I can patch the run.sh file (around line 8) to shift the contents to the solution folder

...
cp -R /opt/test-runner/elm-home $2
export HOME=$2/elm-home
...

But then I get this:

$ docker run --mount type=bind,src=`pwd`/leap,dst=/solution --mount type=bind,src=`pwd`/out,dst=/output -e HOME=/opt/test-runner/elm-home --network none --read-only elm-test-runner leap /solution /output
leap, /solution, /output
Running tests
-- PROBLEM LOADING PACKAGE LIST ------------------------------------------------

I need the list of published packages to verify your dependencies, so I tried to
fetch:

    https://package.elm-lang.org/all-packages

But my HTTP library is giving me the following error message:

    ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "package.elm-lang.org", service name: Just "443"): does not exist (Try again)

Are you somewhere with a slow internet connection? Or no internet? Does the link
I am trying to fetch work in your browser? Maybe the site is down? Does your
internet connection have a firewall that blocks certain domains? It is usually
something like that!

Compilation failed while attempting to build /solution/tests/Tests.elm
Converting Junit output to exercism output
Traceback (most recent call last):
  File "/opt/test-runner/bin/process_results.py", line 60, in <module>
    convert_junitxml_to_json(opts.xml_path, opts.output_path)
  File "/opt/test-runner/bin/process_results.py", line 45, in convert_junitxml_to_json
    tree = ET.parse(xml_path)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 598, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
Finished

[ccare]

Actually, it just occurred to me that I could relax the read-only requirement and just test the network. If I do this, I can reliably see the similar error about network

$ docker run --mount type=bind,src=`pwd`/leap,dst=/solution --mount type=bind,src=`pwd`/out,dst=/output -e HOME=/opt/test-runner/elm-home --network none  elm-test-runner leap /solution /output
leap, /solution, /output
Running tests
-- PROBLEM LOADING PACKAGE LIST ------------------------------------------------

I need the list of published packages to verify your dependencies, so I tried to
fetch:

    https://package.elm-lang.org/all-packages

But my HTTP library is giving me the following error message:

    ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 0, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "package.elm-lang.org", service name: Just "443"): does not exist (Try again)

Are you somewhere with a slow internet connection? Or no internet? Does the link
I am trying to fetch work in your browser? Maybe the site is down? Does your
internet connection have a firewall that blocks certain domains? It is usually
something like that!

Compilation failed while attempting to build /solution/tests/Tests.elm
Converting Junit output to exercism output
Traceback (most recent call last):
  File "/opt/test-runner/bin/process_results.py", line 60, in <module>
    convert_junitxml_to_json(opts.xml_path, opts.output_path)
  File "/opt/test-runner/bin/process_results.py", line 45, in convert_junitxml_to_json
    tree = ET.parse(xml_path)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 598, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
Finished

[ccare]

^ Hope those comments shed some light on what I'm seeing. It's very odd that it appears to be working for you. I feel that I'm missing something obvious.

Thanks,

Charles

[ceddlyburge]

Hi Charles

I am on Docker version 19.03.5, build 633a0ea, which seems close enough.

I think the main difference is that I am building the Docker image on windows, and you are building it on Mac.

It definitely works on my computer without the network, so I think this is a probably a red herring.

The file system permissions seem a more likely problem, especially as this is written to the output when I build the Docker image. I had assumed that the --read-only flag would override these permissions, but maybe not.

Successfully tagged elm-test-runner:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

I have saved the image that I create (docker save elm-test-runner > elm-test-runner.tar) and have emailed it to you (its too big to attach here). Please can you try using this image and let me know how you get on? You can use docker load for this I think.

Thanks, Cedd

Please try using this image and see if you still get the same problem.

Thanks, Cedd

[ccare]

I've 1..1'd you on slack, but it seem that I get the same output as I described before with that image.

:elm-test-runner ccare$ docker run --mount type=bind,src=`pwd`/leap,dst=/solution --mount type=bind,src=`pwd`/out,dst=/output -e HOME=/opt/test-runner/elm-home --network none --read-only elm-test-runner:latest leap /solution /output
leap, /solution, /output
Running tests
elm: /opt/test-runner/elm-home/.elm: createDirectory: permission denied (Read-only file system)
-- ERROR -----------------------------------------------------------------------
I ran into something that bypassed the normal error reporting process! I
extracted whatever information I could from the internal error:
>   thread blocked indefinitely in an MVar operation
These errors are usually pretty confusing, so start by asking around on one of
forums listed at https://elm-lang.org/community to see if anyone can get you
unstuck quickly.
-- REQUEST ---------------------------------------------------------------------
If you are feeling up to it, please try to get your code down to the smallest
version that still triggers this message. Ideally in a single Main.elm and
elm.json file.
From there open a NEW issue at https://github.com/elm/compiler/issues with your
reduced example pasted in directly. (Not a link to a repo or gist!) Do not worry
about if someone else saw something similar. More examples is better!
This kind of error is usually tied up in larger architectural choices that are
hard to change, so even when we have a couple good examples, it can take some
time to resolve in a solid way.elm: thread blocked indefinitely in an MVar operation
Compilation failed while attempting to build /solution/tests/Tests.elm
Converting Junit output to exercism output
Traceback (most recent call last):
  File "/opt/test-runner/bin/process_results.py", line 60, in <module>
    convert_junitxml_to_json(opts.xml_path, opts.output_path)
  File "/opt/test-runner/bin/process_results.py", line 45, in convert_junitxml_to_json
    tree = ET.parse(xml_path)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 1197, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.7/xml/etree/ElementTree.py", line 598, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: no element found: line 1, column 0
Finished

[mpizenberg]

I think I've figured out all issues, I still have few things to verify but I'll get back to you soon. elm_home.tar.gz

[mpizenberg]

Ok I've setup branch remote-elm-home with a config that should work in read only and without network. https://github.com/exercism/elm-test-runner/tree/remote-elm-home

For that to work in --read-only, I mounted the /tmp to the /tmp dir and used the tmp dir as writable working directory both for the compilation and for elm home since it needs to use a lock file. The results.json is moved at the end to the output directory.

For the --network none to work it was partially my fault. The provided elm home I gave you only contained the most recent versions of the potential packages needed. However a project spec in elm.json does not use lower bounds but exact versions of packages. Once I added all previous possible versions it seems to work now.

However the directory grew to 8MB so I preferred using curl in docker build and not store this in the repo. I've setup the modifications in another branch starting at a commit previous to the one adding elm home files. So if everything works and we have no commit constraints on your side yet @ccare we could reset hard master at remote-elm-home and get rid of those commits in the history.

Let us know if everything works as intended.

PS I haven't updated the windows script @ceddlyburge. I'll let you handle that since I don't have any windows machine.

[ccare]

It's looking a lot better. We will need those two new env-vars put in the run.sh script though.

You can write to the input folder if you want (instead of sideways copying everything) but /tmp is great for now.

So I added these into the script

export ELM_HOME=/opt/test-runner/.elm
export CONTEXT=test-runner \

And, once I did that, I got this...

Copying exercise files into /tmp/exercism-elm/a
Running tests
elm: /tmp/exercism-elm/a/.elm/0.19.1/packages/lock: openFd: permission denied (Permission denied)

-- ERROR -----------------------------------------------------------------------

I ran into something that bypassed the normal error reporting process! I
extracted whatever information I could from the internal error:

>   thread blocked indefinitely in an MVar operation

These errors are usually pretty confusing, so start by asking around on one of
forums listed at https://elm-lang.org/community to see if anyone can get you
unstuck quickly.

-- REQUEST ---------------------------------------------------------------------

If you are feeling up to it, please try to get your code down to the smallest
version that still triggers this message. Ideally in a single Main.elm and
elm.json file.

From there open a NEW issue at https://github.com/elm/compiler/issues with your
reduced example pasted in directly. (Not a link to a repo or gist!) Do not worry
about if someone else saw something similar. More examples is better!

This kind of error is usually tied up in larger architectural choices that are
hard to change, so even when we have a couple good examples, it can take some
time to resolve in a solid way.elm: thread blocked indefinitely in an MVar operation
Compilation failed while attempting to build /tmp/exercism-elm/a/tests/Tests.elm
/opt/test-runner # cd  ../test-runner/^C

Adding a chmod -R u+rw $TMP_DIR/.elm after the copy to /tmp seems to make that better.

Then I just get the output

Copying exercise files into /tmp/exercism-elm/a
Running tests

Although no results.json as far as I can see. Any thoughts?

[mpizenberg]

If I put them inside run.sh, this script will not be runnable anymore outside of docker / test-runner. I guess that's ok since that is the objective anyway. In that case, the CONTEXT env var is not required anymore. We can just always do what was inside that guard. Just ELM_HOME need to be set inside run.sh.

Do you see a way to still have run.sh usable outside docker Cedd?

On Mon, Jan 27, 2020, 12:01 Charles Care notifications@github.com wrote:

It's looking a lot better. We will need those two new env-vars put in the run.sh script

fba7a9e#diff-e5794cc18154d40311c02993db06baa0R44 https://github.com/exercism/elm-test-runner/commit/fba7a9ed11ffb592e0862efa511f478e1c62a9a1#diff-e5794cc18154d40311c02993db06baa0R44

fba7a9e#diff-e5794cc18154d40311c02993db06baa0R45 https://github.com/exercism/elm-test-runner/commit/fba7a9ed11ffb592e0862efa511f478e1c62a9a1#diff-e5794cc18154d40311c02993db06baa0R45

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=AAWFOCKODRIWVKUPI3IBTUDQ725J3A5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ7DMQY#issuecomment-578696771, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWFOCKMMKHQC5272LJNA23Q725J3ANCNFSM4KHCBC7Q .

[mpizenberg]

Charles, is there a way to detect that we are inside the test-runner? Any environment variable set on your side?

On Mon, Jan 27, 2020, 12:52 Matthieu Pizenberg matthieu.pizenberg@gmail.com wrote:

If I put them inside run.sh, this script will not be runnable anymore outside of docker / test-runner. I guess that's ok since that is the objective anyway. In that case, the CONTEXT env var is not required anymore. We can just always do what was inside that guard. Just ELM_HOME need to be set inside run.sh.

Do you see a way to still have run.sh usable outside docker Cedd?

On Mon, Jan 27, 2020, 12:01 Charles Care notifications@github.com wrote:

It's looking a lot better. We will need those two new env-vars put in the run.sh script

fba7a9e#diff-e5794cc18154d40311c02993db06baa0R44 https://github.com/exercism/elm-test-runner/commit/fba7a9ed11ffb592e0862efa511f478e1c62a9a1#diff-e5794cc18154d40311c02993db06baa0R44

fba7a9e#diff-e5794cc18154d40311c02993db06baa0R45 https://github.com/exercism/elm-test-runner/commit/fba7a9ed11ffb592e0862efa511f478e1c62a9a1#diff-e5794cc18154d40311c02993db06baa0R45

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=AAWFOCKODRIWVKUPI3IBTUDQ725J3A5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ7DMQY#issuecomment-578696771, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAWFOCKMMKHQC5272LJNA23Q725J3ANCNFSM4KHCBC7Q .

[ceddlyburge]

Hi Matthieu, I'm not too worried about being able to use run.sh outside of the docker environment. We have run-in-docker.sh which allows for local use / testing.

Cheers, Cedd

On Mon, Jan 27, 2020 at 11:59 AM Matthieu Pizenberg < notifications@github.com> wrote:

Charles, is there a way to detect that we are inside the test-runner? Any environment variable set on your side?

On Mon, Jan 27, 2020, 12:52 Matthieu Pizenberg < matthieu.pizenberg@gmail.com> wrote:

If I put them inside run.sh, this script will not be runnable anymore outside of docker / test-runner. I guess that's ok since that is the objective anyway. In that case, the CONTEXT env var is not required anymore. We can just always do what was inside that guard. Just ELM_HOME need to be set inside run.sh.

Do you see a way to still have run.sh usable outside docker Cedd?

On Mon, Jan 27, 2020, 12:01 Charles Care notifications@github.com wrote:

It's looking a lot better. We will need those two new env-vars put in the run.sh script

fba7a9e#diff-e5794cc18154d40311c02993db06baa0R44 < https://github.com/exercism/elm-test-runner/commit/fba7a9ed11ffb592e0862efa511f478e1c62a9a1#diff-e5794cc18154d40311c02993db06baa0R44

fba7a9e#diff-e5794cc18154d40311c02993db06baa0R45 < https://github.com/exercism/elm-test-runner/commit/fba7a9ed11ffb592e0862efa511f478e1c62a9a1#diff-e5794cc18154d40311c02993db06baa0R45

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=AAWFOCKODRIWVKUPI3IBTUDQ725J3A5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ7DMQY#issuecomment-578696771 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAWFOCKMMKHQC5272LJNA23Q725J3ANCNFSM4KHCBC7Q

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/exercism/elm-test-runner/issues/1?email_source=notifications&email_token=ADHR5R673G6EKHYL4UEHWE3Q73EBDA5CNFSM4KHCBC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ7IAOA#issuecomment-578715704, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHR5RZH45JIC4GKI2I5ILLQ73EBDANCNFSM4KHCBC7Q .

[ccare]

Yes, we need run.sh to default to assuming it's in prod.

[mpizenberg]

Ok change done in 4c189a433. If everything works right I'll reset master on this commit to remove from history the packages cache we added in master.

[ccare]

Thanks! We're so close now.

The two changes in https://github.com/exercism/elm-test-runner/pull/2 will complete the picture. The chmod allows elm to lock dependencies, and the relaxing of -e helps when running solutions that fail tests.

[ccare]

Thanks for all of the help. The test runner now works. I've raised a separate issue regarding failure on compilation errors (https://github.com/exercism/elm-test-runner/issues/3) but I'll close this one now.

Thanks again!