Unpacking inner tarball failed (Windows, NTFS, WSL with options=metadata)

[nicbet]

Observed behaviour:

mix phx.new myapp
* creating myapp/config/config.exs
* creating myapp/config/dev.exs
* creating myapp/config/prod.exs
* creating myapp/config/prod.secret.exs
* creating myapp/config/test.exs
* creating myapp/lib/myapp/application.ex
* creating myapp/lib/myapp.ex
* creating myapp/lib/myapp_web/channels/user_socket.ex
* creating myapp/lib/myapp_web/views/error_helpers.ex
* creating myapp/lib/myapp_web/views/error_view.ex
* creating myapp/lib/myapp_web/endpoint.ex
* creating myapp/lib/myapp_web/router.ex
* creating myapp/lib/myapp_web.ex
* creating myapp/mix.exs
* creating myapp/README.md
* creating myapp/test/support/channel_case.ex
* creating myapp/test/support/conn_case.ex
* creating myapp/test/test_helper.exs
* creating myapp/test/myapp_web/views/error_view_test.exs
* creating myapp/lib/myapp_web/gettext.ex
* creating myapp/priv/gettext/en/LC_MESSAGES/errors.po
* creating myapp/priv/gettext/errors.pot
* creating myapp/lib/myapp/repo.ex
* creating myapp/priv/repo/seeds.exs
* creating myapp/test/support/data_case.ex
* creating myapp/lib/myapp_web/controllers/page_controller.ex
* creating myapp/lib/myapp_web/templates/layout/app.html.eex
* creating myapp/lib/myapp_web/templates/page/index.html.eex
* creating myapp/lib/myapp_web/views/layout_view.ex
* creating myapp/lib/myapp_web/views/page_view.ex
* creating myapp/test/myapp_web/controllers/page_controller_test.exs
* creating myapp/test/myapp_web/views/layout_view_test.exs
* creating myapp/test/myapp_web/views/page_view_test.exs
* creating myapp/.gitignore
* creating myapp/assets/brunch-config.js
* creating myapp/assets/css/app.css
* creating myapp/assets/css/phoenix.css
* creating myapp/assets/js/app.js
* creating myapp/assets/js/socket.js
* creating myapp/assets/package.json
* creating myapp/assets/static/robots.txt
* creating myapp/assets/static/images/phoenix.png
* creating myapp/assets/static/favicon.ico

Fetch and install dependencies? [Yn] y
* running mix deps.get
Resolving Hex dependencies...
Dependency resolution completed:
New:
  connection 1.0.4
  cowboy 1.1.2
  cowlib 1.0.2
  db_connection 1.1.3
  decimal 1.5.0
  ecto 2.2.11
  file_system 0.2.6
  gettext 0.16.0
  mime 1.3.0
  phoenix 1.3.4
  phoenix_ecto 3.6.0
  phoenix_html 2.12.0
  phoenix_live_reload 1.1.7
  phoenix_pubsub 1.1.1
  plug 1.7.1
  plug_crypto 1.0.0
  poison 3.1.0
  poolboy 1.5.1
  postgrex 0.13.5
  ranch 1.3.2
* Updating phoenix (Hex package)
** (Mix) Unpacking tarball failed: inner tarball error, invalid argument

Expected behaviour: dependencies fetched successfully.

Note that the issue also manifests as:

mix new testapp
* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/testapp.ex
* creating test
* creating test/test_helper.exs
* creating test/testapp_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd testapp
    mix test

Run "mix help" for more commands.

cd testapp

iex -S mix
Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Compiling 1 file (.ex)
** (File.Error) could not touch "/mnt/d/Projects/Elixir/testapp/_build/dev/lib/testapp/ebin/Elixir.Testapp.beam": invalid argument
    (elixir) lib/file.ex:528: File.touch!/2
    (mix) lib/mix/compilers/elixir.ex:561: anonymous fn/4 in Mix.Compilers.Elixir.write_manifest/5
    (elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
    (mix) lib/mix/compilers/elixir.ex:555: Mix.Compilers.Elixir.write_manifest/5
    (mix) lib/mix/compilers/elixir.ex:198: Mix.Compilers.Elixir.compile_manifest/9
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/tasks/compile.all.ex:68: Mix.Tasks.Compile.All.run_compiler/2
    (mix) lib/mix/tasks/compile.all.ex:52: Mix.Tasks.Compile.All.do_compile/4

[OvermindDL1]

@nicbet As a windows user, what would you do to fix windows giving an invalid access error upon accessing a file?

[nicbet]

For what it's worth I've been tracing the code by hand and here is something interesting:

elixir_utils calls file:write_file_info(Name, #file_info{mtime=Time}, [{time, universal}]).

On Windows, if I simulate this call like this:

defmodule XFile do
  require Record
  Record.defrecord(:file_info,  Record.extract(:file_info, from_lib: "kernel/include/file.hrl"))
end

require XFile

finfo = XFile.file_into(mtime: :calendar.universal_time())

which return me a

{:file_info, :undefined, :undefined, :undefined, :undefined,
 {{2018, 11, 1}, {17, 21, 20}}, :undefined, :undefined, :undefined, :undefined,
 :undefined, :undefined, :undefined, :undefined}

and then calling

:file.write_file_info("test.txt", finfo)

I get exactly the same error {:error, :einval}. On Mac OS X, the same code results in an :ok

I can get the call in Windows to succeed if I specify the file info fully:

finfo = {:file_info, 0, :regular, :read_write, {{2018, 11, 1}, {13, 3, 27}},
 {{2018, 11, 1}, {13, 4, 28}}, {{2018, 11, 1}, {13, 4, 28}}, 33188, 1, 13, 0,
 449, 1000, 1000}

:file.write_file_info("test.txt", finfo)
:ok

[nicbet]

it appears that if uid and gid are :undefined, :prim_file.write_file_info_1 calls an internal set_owner which sets uid and gid to -1 which causes the {:error, :einval}:

finfo = {:file_info, 0, :regular, :read_write, {{2018, 11, 1}, {13, 3, 27}},
 {{2018, 11, 1}, {13, 4, 28}}, {{2018, 11, 1}, {13, 4, 28}}, 33188, 1, 13, 0,
 449, -1, -1}

:file.write_file_info("test.txt", finfo)
{:error, :einval}

[ericmj]

@nicbet Thank you for investigating this.

Do we know if it's the correct behavior to set uid/gid to -1? It was my understanding that OTP would only modify the properties where the value != :undefined.

[nicbet]

It's defined in https://github.com/erlang/otp/blob/master/erts/preloaded/src/prim_file.erl#L631

[nicbet]

The plot thickens, ... a small C program:

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>

int main() {
        char path[] = "test.txt";
        int ret = chown(path, -1, -1);

        if (ret != 0) {
                perror("chown");
                printf("\n Error val: %d\n", errno);
        }

        return ret;
}

On Mac OS X returns 0, on Windows returns errno of 22 which is EINVAL.

So apparently WSL does not support "-1" for either gid or uid, even though the spec (http://pubs.opengroup.org/onlinepubs/7908799/xsh/chown.html) says If owner or group is specified as (uid_t)-1 or (gid_t)-1 respectively, the corresponding ID of the file is unchanged.

[OvermindDL1]

So sounds like a WSL bug then indeed, hrmm...

[nicbet]

I think it's also worth mentioning that for anyone who encounters this problem, for now be aware that this issue only manifests on mounted (DrvFS) file systems.

You can still do you Elixir development on your root filesystem (cd ~).

[palmo25]

So that means that at the present it ‘ s not possibile to use Elixir and Docker on WSL correct? I think this is quite a big issue...Any hope that Microsoft will address it?

[nicbet]

From what I've seen reading through the open bug reports over on the WSL GitHub, it appears that the nodejs/yarn community is hitting a similar, possibly the same bug, which appears to have been fixed in Windows 1809. I was looking to update to see if 1809 fixes the issue, but alas MSFT has for now stopped rolling out the 1809 upgrade. When 1809 rolls out and the issue persists, I think we should open new bug over at WSL.

[palmo25]

Ok thanks a lot for your feedback. I’ll go back to local development while keeping an eye to the Windows update.

[nicbet]

Good news! I just updated to Windows 10 build 1809, and the issue seems to be fixed.

The C program, as well as Erlang/OTPs chown call with -1 as uid now work correctly and return 0.

[AntJanus]

I just wanted to jump in, I'm having the same issue. I'm on NTFS but I'm not using WSL.

[christianjgreen]

I am also getting this issues on both windows 1803 and 1809 not using WSL.

[ericmj]

We have noticed tarballs with symlinks can have issues on Windows so that is worth looking into for people have have this issue. The Hex team cannot reproduce this issue so we are closing it for now. If you have more information please post it here or open a new issue. Thank your for the report!