New error: `can not merge projects`

[goretkin]

In the Julia 1.4 rc images, I am getting a new error from Pkg.jl. You can see it here:

https://gitlab.com/ExampleJuliaGroup/ExamplePackage.jl/pipelines/122968576

   [8dfed614] + Test 
    Testing ExamplePackage
ERROR: can not merge projects
Stacktrace:
 [1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:53
 [2] (::Pkg.Operations.var"#84#88"{Pkg.Operations.var"#95#96"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{String,1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Nothing,String})(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1349
 [3] mktempdir(::Pkg.Operations.var"#84#88"{Pkg.Operations.var"#95#96"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{String,1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Nothing,String}, ::String; prefix::String) at ./file.jl:682
 [4] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [5] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1324
 [6] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1488
 [7] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:316
 [8] #test#68 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:297 [inlined]
 [9] #test#65 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:294 [inlined]
 [10] top-level scope at none:4

The error arises after, in the context of the project, I do:

pkg"activate test"; pkg"dev ."; pkg"update"; pkg"activate ."

with the purpose of having the test/Manifest.toml reflect all the dependencies of the project (i.e. includes the information in ./Manifest.toml)

If there is a better way to accomplish that, I'd be happy to know. In any case, it's at least a backwards compatibility bug.

This is the PR that introduced this error message. PR: https://github.com/JuliaLang/Pkg.jl/pull/1427/files

[00vareladavid]

1707 will probably solve this.

You are seeing this error because of the explicit pkg"dev .". Can you try to re-run your tests without that? The end goal is for Pkg to automatically handle "sync" between the root project and the test project.

[goretkin]

Yes, I isolated it to the pkg"dev .", but I indeed want to express the exact dependencies in the test/Manifest.toml. Is there another way to do that?

By the way, I got this idea from https://discourse.julialang.org/t/nested-environments-in-package/33157

[00vareladavid]

The package is already tracked automatically when the test project is instantiated. Consider a brand new package:

(Foo) pkg> test
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Updating `/tmp/Foo/Project.toml`
 [no changes]
   Updating `/tmp/Foo/Manifest.toml`
 [no changes]
    Testing Foo
Status `/tmp/jl_6AyQwN/Manifest.toml`
  [8c2ffe6e] Foo v0.1.0 [`/tmp/Foo`]  # <- PACKAGE TRACKED BY PATH
  [2a0f44e3] Base64 
  [8ba89e20] Distributed 
  [b77e0a4c] InteractiveUtils 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [9a3f8284] Random 
  [9e88b42a] Serialization 
  [6462fe0b] Sockets 
  [8dfed614] Test 
    Testing Foo tests passed

You can see that Foo automatically shows up as a dependency being tracked by path.

If you add a new dependency, the test project will autmatically reflect that:

(Foo) pkg> add Example
  Resolving package versions...
   Updating `/tmp/Foo/Project.toml`
  [7876af07] + Example v0.5.3
   Updating `/tmp/Foo/Manifest.toml`
  [7876af07] + Example v0.5.3

(Foo) pkg> test
    Testing Foo
Status `/tmp/jl_yqXG9W/Manifest.toml`
  [7876af07] Example v0.5.3 # <- NEW DEPENDENCY
  [8c2ffe6e] Foo v0.1.0 [`/tmp/Foo`]
  [2a0f44e3] Base64 
  [8ba89e20] Distributed 
  [b77e0a4c] InteractiveUtils 
  [56ddb016] Logging 
  [d6f4376e] Markdown 
  [9a3f8284] Random 
  [9e88b42a] Serialization 
  [6462fe0b] Sockets 
  [8dfed614] Test 
    Testing Foo tests passed 

What do you hope to accomplish with pkg"dev ."? How does that differ from the current Pkg.test behavior?

[00vareladavid]

Phrased differently: why are you trying to do something manually that Pkg does automatically for you? Not trying to be defensive, just trying to understand your POV :slightly_smiling_face:

[goretkin]

What about /tmp/jl_yqXG9W/test/Manifest.toml, does it track the dependency? From what I can tell, the answer is no.

I don't want to check in a $library/Manifest.toml, but I do want to check in a $library/test/Manifest.toml.

Hope I understood your example.

[00vareladavid]

Yes, it tracks the dependency. This is why Example appears the second time test is run.

Why do you want to check in $library/test/Manifest.toml?

[goretkin]

I'm probably using "tracks the dependency" incorrectly. Can you confirm whether Foo/test/Manifest.toml contains an entry for Example?

The reason it doesn't make sense to check in $library/Manifest.toml is because when some other code does import $library, then $library/Manifest.toml is ignored.

But I want to check in $library/test/Manifest.toml because I view the tests as an application. When the tests run, it makes sense to control the environment they're running in, and $library/test/Manifest.toml accomplishes this perfectly.

Even though https://julialang.github.io/Pkg.jl/v1/glossary/ says "Applications can provide global configuration, whereas packages cannot." it's not exactly true, because a package should have a test application within it, which can provide a global configuration.

I wrote this in an answer a few weeks ago: https://stackoverflow.com/questions/58996197/when-should-a-julia-project-have-a-manifest-and-project-file I would happy to hear other thoughts on it.

[goretkin]

So does it seem like my desired workflow is not really compatible with the recent changes in Pkg?

To summarize, for a given library, I want

• not to check in Manifest.toml
• to check in test/Manifest.toml
• (have the option to) run the tests in a the environment specified by test/Manifest.toml

[tkf]

I have the same preference (and commented it in https://github.com/JuliaLang/Pkg.jl/issues/1233#issuecomment-572301060). Since Pkg.jl does not support this workflow, I moved test/Manifest.toml to test/environments/main/Manifest.toml (also using test/environments/jl10/Manifest.toml for Julia 1.0). Here is my typical setup: https://github.com/tkf/BangBang.jl/blob/91dfa73d587deee4b7047e80b968d0b5267ca405/.travis.yml#L17-L23

The error arises after, in the context of the project, I do:

pkg"activate test"; pkg"dev ."; pkg"update"; pkg"activate ."

with the purpose of having the test/Manifest.toml reflect all the dependencies of the project (i.e. includes the information in ./Manifest.toml)

FYI, you can do pkg> dev .. at test directory and check in the resulting test/Manifest.toml. It can be used in other machines since it only contains the relative path.

[tbonza]

This may not be best practices -- I deleted the test/Manifest.toml file and moved on. It was regenerated the next time I ran my test suite

[goretkin]

do pkg> dev .. at test directory

@tkf , Can you clarify why and if that's different from pkg"activate test"; pkg"dev ."; pkg"update"; pkg"activate ." done from the project directory? I didn't notice a difference.

[tkf]

I think pkg"activate test"; pkg"dev ."; is OK too since it writes a relative path to test/Manifest.toml. My point was rather that you can check in test/Manifest.toml if you make sure that it contains only relative paths. This way, you can run the tests just by activating test and your test result is always reproducible (though maybe that's not what you want since you are running pkg"update" explicitly).

[goretkin]

I still feel completely lost about what to do with these Manifests. I have a project that looks like

- MainProject
    Project.toml
  - src
      a.jl ...
  - test
       runtests.jl
       Project.toml # test-specific dependencies
       Manifest.toml # reproducible environment for running tests

And when I do Pkg.test(), a MainProject/Manifest.toml is created and then I get a can not merge error.

What do I do if I want to check in the environment that my tests passed in? It would leave a bad taste in my mouth to check in generated MainProject/Manifest.toml, but in any case that is not enough because the tests-specific dependencies do not appear to get recorded there.

[KristofferC]

Dup of https://github.com/JuliaLang/Pkg.jl/issues/1585