Packaging for NixOS

[CMCDragonkai]

I'm currently working on packaging for NixOS: https://github.com/NixOS/nixpkgs/issues/49438

I may ask for help regarding the integration.

[efiop]

Hi @CMCDragonkai !

Great stuff! Thank you for your work! Please don't hesitate to ask any questions, we'll be happy to help 🙂

[efiop]

@CMCDragonkai Also note that we have binary packages as well. For example, our deb and rpm packages contain dvc built with pyinstaller with all of the possible dependencies, so it doesn't rely on anything on the system and carries all the libs(including python itself) with it. You should be able to utilize that as well, if you are not keen on packaging every pip dependency separately. See scripts/build_posix.sh for more info.

[CMCDragonkai]

Nix prefers to get everything built from source barring proprietary applications.

[CMCDragonkai]

I've packaged all the dependencies for dvc into NixOS. Now I'm onto packaging dvc itself. However your pip installation doesn't have a source distribution. NixOS prefers to build things from the source. Can you push up your source distributions to PyPi as well? https://pypi.org/project/dvc/#files

Note that source distribution should contain any tests for dvc as well.

[ghost]

@CMCDragonkai , what if instead of fetching from pypi you set something like:

src = fetchurl {
  url = "https://github.com/iterative/dvc/archive/${pkgver}.tar.gz";
  sha256 = "sha of the tar.gz";
};

I don't know if it's possible, but that's what I'm currently doing for the Arch build: PKGBUILD.

However, I think it is a good idea to release the source code to pypi

[ghost]

By the way, great work so far, @CMCDragonkai ! I noticed that you also packaged th missing dependencies on nixpkgs :+1: ! Regarding your question about the optional dependencies, I've seen both enableX and supportX (e.g. supportSSH)

NOTE: remember to include HDFS as a remote

[CMCDragonkai]

I can use fetchFromGithub instead that's usually the standard as long as Github releases are considered as official as PyPI releases then it's fine.

[ghost]

Yes, you can rely on GitHub releases, as far as I know, we are supporting them :smiley:

[CMCDragonkai]

Ok building from source results in:

»» ~/Projects/nixpkgs
 ♜ nix-build ./default.nix --attr 'dvc'      (dvc) pts/8 17:30:56
these derivations will be built:
  /nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv
building '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv'...
unpacking sources
unpacking source archive /nix/store/wj4qqj4qvinvrgm1njq65klfdh6ngzx6-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/utils/httpd.py
patching sources
configuring
building
Traceback (most recent call last):
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 83, in <module>
    refresh()
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 73, in refresh
    if not Git.refresh(path=path):
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/cmd.py", line 290, in refresh
    raise ImportError(err)
ImportError: Bad git executable.
The git executable must be specified in one of the following ways:
    - be included in your $PATH
    - be set via $GIT_PYTHON_GIT_EXECUTABLE
    - explicitly set via git.refresh()

All git commands will error until this is rectified.

This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
    - quiet|q|silence|s|none|n|0: for no warning or exception
    - warn|w|warning|1: for a printed warning
    - error|e|raise|r|2: for a raised exception

Example:
    export GIT_PYTHON_REFRESH=quiet

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 5, in <module>
    from dvc import VERSION
  File "/build/source/dvc/__init__.py", line 53, in <module>
    __version__ = _update_version_file()
  File "/build/source/dvc/__init__.py", line 20, in _update_version_file
    from git import Repo
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 85, in <module>
    raise ImportError('Failed to initialize: {0}'.format(exc))
ImportError: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
    - be included in your $PATH
    - be set via $GIT_PYTHON_GIT_EXECUTABLE
    - explicitly set via git.refresh()

All git commands will error until this is rectified.

This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
    - quiet|q|silence|s|none|n|0: for no warning or exception
    - warn|w|warning|1: for a printed warning
    - error|e|raise|r|2: for a raised exception

Example:
    export GIT_PYTHON_REFRESH=quiet

builder for '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv' failed with exit code 1
error: build of '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv' failed

The tests require the existence of git. So I add that into buildInputs.

Furthermore the tests also require nosetests. But not available.

[CMCDragonkai]

It appears that test dependencies need quite a lot more stuff: https://github.com/iterative/dvc/blob/master/test-requirements.txt

I'd like to know more about your test enviornment. Even after adding all the dependencies, the first error I get is:

Error: Failed to load dir cache '../../b/uild/dvc-test.4363.yk9hglkp.0d76514b-9d20-44bd-a35e-0abe6b4b2720/.dvc/cache/12/3.dir' - [Errno 2] No such file or directory: '/b/uild/dvc-test.4363.yk9hglkp.0d76514b-9d20-44bd-a35e-0abe6b4b2720/.dvc/cache/12/3.dir'

[CMCDragonkai]

I just realised that some of the tests call awscli which means it relies on test AWS services. Does this mean the tests are meant to be executed with DVC's private testing servers? In that case is there a way to only test dvc with unattended tests? If not, I'll have to disable tests.

[CMCDragonkai]

Another question, is that when I have finally built the dvc script. Running the script results in this message saying:

/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7: No module named dvc

[ghost]

For simplicity, @CMCDragonkai, I would recommend you to disable the tests (we are already testing the code on our CI, so we are covered), we can enable them after fixing those dependency issues.

About the No module named dvc error, is hard to tell :thinking: Do you have more details about it? What about running a shell and import dvc

[CMCDragonkai]

The PR for Nixpkgs is here: https://github.com/NixOS/nixpkgs/pull/54530

I've disabled all tests. So the last thing to figure out is that No module named dvc.

[CMCDragonkai]

@mroutis That error is quite strange. It's an essentially an import error. However since dvc is imported everywhere obviously it can be imported otherwise the the whole thing should fail. Since it only reports it at the end. Maybe there's something that gets executed at the end that tries to import dvc in some special way and fails to do so?

And yes it's perfectly possible to import dvc.

[ghost]

@CMCDragonkai, if you start a Python interpreter and enter import dvc, does it works? I don't know if paths should be updated or if the shell has enough information to look at the package;

I would give it a try tomorrow :)

[CMCDragonkai]

Yes it does inside the build environment. But dvc is installed as an application, not a library. So my normal python interpreter won't be able to find it. NixOS considers apps and libraries separately. Hmm I'm wondering what is happening is that something inside dvc script calls python interpreter from the outside and imports dvc, if thats true then it would fail.

[ghost]

@CMCDragonkai , possibly bin/dvc?

#!/bin/bash

PYTHONPATH=$DVC_HOME python -m dvc $@

[CMCDragonkai]

That's ignored by the Nix expression because you are using setup.py which doesn't reference that script. Instead it only uses this:

https://github.com/iterative/dvc/blob/master/setup.py#L73-L75

Or the scripts setting in setup.py which you don't have.

So it shouldn't be that bin/dvc script. In fact that script does not even exist at the build outputs.

[efiop]

@CMCDragonkai That is correct. It is hard to point out to the specific thing that had broke, since I'm not familiar with NixOS packages. I would take a look at other similar tools written in python and nixos packages for them. For example https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/admin/awscli/default.nix . Did you use a similar project as a template or did you create it from scratch?

[CMCDragonkai]

@efiop Yes, here's the PR: https://github.com/NixOS/nixpkgs/pull/54530/files

It's a very standard python application. I'm suspecting what I'm saying is true, somewhere during the execution of dvc, it tries to import dvc somewhere and that fails, but it doesn't cause a critical failure since the commands still work.

[efiop]

@CMCDragonkai Hm, thinking about it, it is probably our analytics daemon launching https://github.com/iterative/dvc/blob/master/dvc/daemon.py#L71 . Here is where it tries to run sys.executable -m dvc.

[efiop]

We could probably change it up, so that it figures out __main__.py path from __file__ and runs it directly with something like sys.executable /path/to/dvc/__main__.py. Which not only should solve the issue you are having, but also be a more correct way of launching analytics daemon.

[CMCDragonkai]

That might be true. Here's an example:

»» ~/Projects/dvctest
 ♖ dvc add data/Posts.xml.zip                 pts/11 11:54:08
/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7: No module named dvc
Adding 'data/Posts.xml.zip' to 'data/.gitignore'.
Saving 'data/Posts.xml.zip' to cache '.dvc/cache'.
Saving information to 'data/Posts.xml.zip.dvc'.

To track the changes with git run:

        git add data/.gitignore data/Posts.xml.zip.dvc
/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7: No module named dvc

See there's one report at the beginning and one report at the end.

[CMCDragonkai]

Is dvc daemon something that gets launched in the background and continues to run between dvc invocations?

[efiop]

@CMCDragonkai Not really :slightly_smiling_face: It is kinda of a misleading name for what is actually a single-run worker, that sends analytics and then just dies.

Created https://github.com/iterative/dvc/issues/1525 .

[efiop]

@CMCDragonkai I've submitted a PR https://github.com/iterative/dvc/pull/1531 that should help with making dvc work properly in your package. We'll release new dvc version with it ASAP, I'll try to notify you here when it is ready, so you could try it out.

[efiop]

@CMCDragonkai 0.24.3 is out. Please give it a try.

[CMCDragonkai]

@efiop Unfortunately that's not sufficient to fix the problem. It is a step in the right direction.

Traceback (most recent call last):                                                                          pts/10 11:48:04
  File "/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages/dvc/__main__.py", line 5, in <module>
    from dvc.main import main
  File "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages/dvc/main.py", line 3, in <module>
    import dvc.logger as logger
  File "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages/dvc/logger.py", line 10, in <module>
    import colorama
ModuleNotFoundError: No module named 'colorama'

Basically after successfully calling dvc daemon, which then tries to call dvc/__main__.py and then dvc/main.py which then calls dvc/logger.py, the first import of a foreign package is import colorama. That fails, because that script is not wrapped with the environmental dependencies that Nix provides. This is because the python environment that Nix produces is a wrapper around the bin script.

See this the package that is built by Nix:

``` . ├── bin │   ├── dvc │   └── .dvc-wrapped ├── lib │   └── python3.7 │   └── site-packages │   ├── dvc │   │   ├── analytics.py │   │   ├── cache.py │   │   ├── cli.py │   │   ├── command │   │   │   ├── add.py │   │   │   ├── base.py │   │   │   ├── cache.py │   │   │   ├── checkout.py │   │   │   ├── config.py │   │   │   ├── daemon.py │   │   │   ├── data_sync.py │   │   │   ├── destroy.py │   │   │   ├── gc.py │   │   │   ├── imp.py │   │   │   ├── __init__.py │   │   │   ├── init.py │   │   │   ├── install.py │   │   │   ├── lock.py │   │   │   ├── metrics.py │   │   │   ├── move.py │   │   │   ├── pipeline.py │   │   │   ├── __pycache__ │   │   │   │   ├── add.cpython-37.pyc │   │   │   │   ├── base.cpython-37.pyc │   │   │   │   ├── cache.cpython-37.pyc │   │   │   │   ├── checkout.cpython-37.pyc │   │   │   │   ├── config.cpython-37.pyc │   │   │   │   ├── daemon.cpython-37.pyc │   │   │   │   ├── data_sync.cpython-37.pyc │   │   │   │   ├── destroy.cpython-37.pyc │   │   │   │   ├── gc.cpython-37.pyc │   │   │   │   ├── imp.cpython-37.pyc │   │   │   │   ├── __init__.cpython-37.pyc │   │   │   │   ├── init.cpython-37.pyc │   │   │   │   ├── install.cpython-37.pyc │   │   │   │   ├── lock.cpython-37.pyc │   │   │   │   ├── metrics.cpython-37.pyc │   │   │   │   ├── move.cpython-37.pyc │   │   │   │   ├── pipeline.cpython-37.pyc │   │   │   │   ├── remote.cpython-37.pyc │   │   │   │   ├── remove.cpython-37.pyc │   │   │   │   ├── repro.cpython-37.pyc │   │   │   │   ├── root.cpython-37.pyc │   │   │   │   ├── run.cpython-37.pyc │   │   │   │   ├── status.cpython-37.pyc │   │   │   │   └── unprotect.cpython-37.pyc │   │   │   ├── remote.py │   │   │   ├── remove.py │   │   │   ├── repro.py │   │   │   ├── root.py │   │   │   ├── run.py │   │   │   ├── status.py │   │   │   └── unprotect.py │   │   ├── config.py │   │   ├── daemon.py │   │   ├── dagascii.py │   │   ├── data_cloud.py │   │   ├── dependency │   │   │   ├── base.py │   │   │   ├── gs.py │   │   │   ├── hdfs.py │   │   │   ├── http.py │   │   │   ├── __init__.py │   │   │   ├── local.py │   │   │   ├── __pycache__ │   │   │   │   ├── base.cpython-37.pyc │   │   │   │   ├── gs.cpython-37.pyc │   │   │   │   ├── hdfs.cpython-37.pyc │   │   │   │   ├── http.cpython-37.pyc │   │   │   │   ├── __init__.cpython-37.pyc │   │   │   │   ├── local.cpython-37.pyc │   │   │   │   ├── s3.cpython-37.pyc │   │   │   │   └── ssh.cpython-37.pyc │   │   │   ├── s3.py │   │   │   └── ssh.py │   │   ├── exceptions.py │   │   ├── __init__.py │   │   ├── istextfile.py │   │   ├── lock.py │   │   ├── logger.py │   │   ├── __main__.py │   │   ├── main.py │   │   ├── output │   │   │   ├── base.py │   │   │   ├── gs.py │   │   │   ├── hdfs.py │   │   │   ├── __init__.py │   │   │   ├── local.py │   │   │   ├── __pycache__ │   │   │   │   ├── base.cpython-37.pyc │   │   │   │   ├── gs.cpython-37.pyc │   │   │   │   ├── hdfs.cpython-37.pyc │   │   │   │   ├── __init__.cpython-37.pyc │   │   │   │   ├── local.cpython-37.pyc │   │   │   │   ├── s3.cpython-37.pyc │   │   │   │   └── ssh.cpython-37.pyc │   │   │   ├── s3.py │   │   │   └── ssh.py │   │   ├── progress.py │   │   ├── project.py │   │   ├── prompt.py │   │   ├── __pycache__ │   │   │   ├── analytics.cpython-37.pyc │   │   │   ├── cache.cpython-37.pyc │   │   │   ├── cli.cpython-37.pyc │   │   │   ├── config.cpython-37.pyc │   │   │   ├── daemon.cpython-37.pyc │   │   │   ├── dagascii.cpython-37.pyc │   │   │   ├── data_cloud.cpython-37.pyc │   │   │   ├── exceptions.cpython-37.pyc │   │   │   ├── __init__.cpython-37.pyc │   │   │   ├── istextfile.cpython-37.pyc │   │   │   ├── lock.cpython-37.pyc │   │   │   ├── logger.cpython-37.pyc │   │   │   ├── __main__.cpython-37.pyc │   │   │   ├── main.cpython-37.pyc │   │   │   ├── progress.cpython-37.pyc │   │   │   ├── project.cpython-37.pyc │   │   │   ├── prompt.cpython-37.pyc │   │   │   ├── stage.cpython-37.pyc │   │   │   ├── state.cpython-37.pyc │   │   │   ├── system.cpython-37.pyc │   │   │   └── updater.cpython-37.pyc │   │   ├── remote │   │   │   ├── azure.py │   │   │   ├── base.py │   │   │   ├── gs.py │   │   │   ├── hdfs.py │   │   │   ├── http.py │   │   │   ├── __init__.py │   │   │   ├── local.py │   │   │   ├── __pycache__ │   │   │   │   ├── azure.cpython-37.pyc │   │   │   │   ├── base.cpython-37.pyc │   │   │   │   ├── gs.cpython-37.pyc │   │   │   │   ├── hdfs.cpython-37.pyc │   │   │   │   ├── http.cpython-37.pyc │   │   │   │   ├── __init__.cpython-37.pyc │   │   │   │   ├── local.cpython-37.pyc │   │   │   │   ├── s3.cpython-37.pyc │   │   │   │   └── ssh.cpython-37.pyc │   │   │   ├── s3.py │   │   │   └── ssh.py │   │   ├── scm │   │   │   ├── base.py │   │   │   ├── git.py │   │   │   ├── __init__.py │   │   │   └── __pycache__ │   │   │   ├── base.cpython-37.pyc │   │   │   ├── git.cpython-37.pyc │   │   │   └── __init__.cpython-37.pyc │   │   ├── stage.py │   │   ├── state.py │   │   ├── system.py │   │   ├── updater.py │   │   └── utils │   │   ├── compat.py │   │   ├── __init__.py │   │   └── __pycache__ │   │   ├── compat.cpython-37.pyc │   │   └── __init__.cpython-37.pyc │   ├── dvc-0.24.3.dist-info │   │   ├── entry_points.txt │   │   ├── INSTALLER │   │   ├── LICENSE │   │   ├── METADATA │   │   ├── RECORD │   │   ├── top_level.txt │   │   └── WHEEL │   └── tests │   ├── unit │   │   ├── daemon.py │   │   ├── __init__.py │   │   ├── __pycache__ │   │   │   ├── daemon.cpython-37.pyc │   │   │   ├── __init__.cpython-37.pyc │   │   │   ├── stage.cpython-37.pyc │   │   │   └── state.cpython-37.pyc │   │   ├── remote │   │   │   ├── azure.py │   │   │   ├── base.py │   │   │   ├── gs.py │   │   │   ├── __init__.py │   │   │   ├── __pycache__ │   │   │   │   ├── azure.cpython-37.pyc │   │   │   │   ├── base.cpython-37.pyc │   │   │   │   ├── gs.cpython-37.pyc │   │   │   │   ├── __init__.cpython-37.pyc │   │   │   │   └── ssh.cpython-37.pyc │   │   │   └── ssh.py │   │   ├── stage.py │   │   └── state.py │   └── utils │   ├── httpd.py │   ├── __init__.py │   └── __pycache__ │   ├── httpd.cpython-37.pyc │   └── __init__.cpython-37.pyc └── nix-support └── propagated-build-inputs 27 directories, 180 files ```

The environment produced by Nix is all specified by `bin/dvc`. Which is:

``` #! /nix/store/vs6d2fjkl4kb3jb7rwibsd76k9v2n4xy-bash-4.4-p23/bin/bash -e export PATH='/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin:/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/bin:/nix/store/x54cj2603lvj6y5fmpkg7x0hvad8kp85-python3.7-setuptools-40.6.2/bin:/nix/store/64pfzf4mcgk8350rk9cmn4rfg0v148ah-python3.7-future-0.17.1/bin:/nix/store/qnzpg1iqbwrjhpjc6h99w328gz31lfv9-python3.7-jsonpath-rw-1.4.0/bin:/nix/store/b8hrpwr27x82zqyvch81zrdv8i2igdxm-python3.7-chardet-3.0.4/bin:/nix/store/4wrg4g7blmb4dd8f9vbg9c5qp4r464bp-python3.7-pyfiglet-0.8.post0/bin:/nix/store/2hzvyknrhys2yl6bw4791w3pfml659fl-python3.7-distro-1.3.0/bin'${PATH:+':'}$PATH export PYTHONNOUSERSITE='true' exec -a "$0" "/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/bin/.dvc-wrapped" "${extraFlagsArray[@]}" "$@" ```

And the called wrapped script is:

``` #!/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7 # -*- coding: utf-8 -*- import sys;import site;import functools;sys.argv[0] = '/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/bin/dvc';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/sc6z8j8qrzkgav31d6dln3x3qpvc3bsz-dvc-0.24.3/lib/python3.7/site-packages','/nix/store/31yh08kpq9hdjyag0429iis36avma2ah-python3.7-ply-3.11/lib/python3.7/site-packages','/nix/store/x54cj2603lvj6y5fmpkg7x0hvad8kp85-python3.7-setuptools-40.6.2/lib/python3.7/site-packages','/nix/store/28n5hxpbqwfpf1mxlk5xlrd8isn0wkyd-python3.7-configparser-3.5.0/lib/python3.7/site-packages','/nix/store/yfa20508fkp32363cdyka87fbw26nffs-python3.7-zc.lockfile-1.4/lib/python3.7/site-packages','/nix/store/vpppyclhskfc79qhnb6qzipglxq7hkcc-python3.7-zope.testing-4.7/lib/python3.7/site-packages','/nix/store/f7lxf28icqxz7kx7j7fp43qcpja20l24-python3.7-zope.interface-4.6.0/lib/python3.7/site-packages','/nix/store/01f9mqlgdqj2plq1flwdgkn08r6bi14w-python3.7-zope.event-4.4/lib/python3.7/site-packages','/nix/store/i8rqa42ci9fma700d9dvcigzzmh6hjya-python3.7-zope.exceptions-4.3/lib/python3.7/site-packages','/nix/store/r6kg1ih7k97z4cdp5r2ab2z2jkd95ii8-python3.7-zope.location-4.2/lib/python3.7/site-packages','/nix/store/bj47fxx4ix1agbqs4q4iqzjxaqr3ywn2-python3.7-zope.proxy-4.3.1/lib/python3.7/site-packages','/nix/store/64pfzf4mcgk8350rk9cmn4rfg0v148ah-python3.7-future-0.17.1/lib/python3.7/site-packages','/nix/store/dfhzqvvr2l7b3cs3l1fs0n197hfarbr1-python3.7-colorama-0.3.9/lib/python3.7/site-packages','/nix/store/gp4vh1czqzm378q9c1fmr7gqnacw2gdr-python3.7-configobj-5.0.6/lib/python3.7/site-packages','/nix/store/ckw5mkgwwhh1j3fn33ysvgicdw8wr4f8-python3.7-six-1.11.0/lib/python3.7/site-packages','/nix/store/pjwlgy35f021mw6bbmvpdi6hjcx6r9v5-python3.7-networkx-2.2/lib/python3.7/site-packages','/nix/store/izdjyd0jkp4mb2pzl67rs1gmy2g68f43-python3.7-decorator-4.3.0/lib/python3.7/site-packages','/nix/store/x5xlhq0gkfjwn8r0w8c1ls43qhwhwxs5-python3.7-PyYAML-3.13/lib/python3.7/site-packages','/nix/store/8zydrhqxlcz39ljx9fplkbs652wws9km-python3.7-GitPython-2.1.11/lib/python3.7/site-packages','/nix/store/iqh4mm2d3yjfc0da78sgbrpjkx847r41-python3.7-gitdb2-2.0.5/lib/python3.7/site-packages','/nix/store/y6hzhdqlba3szsf2k9smp70ypi2fas8h-python3.7-smmap2-2.0.5/lib/python3.7/site-packages','/nix/store/p1kvm1h9d0kc74q3i1nf4c9y6z6nmw4y-python3.7-ddt-1.2.0/lib/python3.7/site-packages','/nix/store/8prggywiwn97glw6svr4spkav9m260vr-python3.7-nanotime-0.5.2/lib/python3.7/site-packages','/nix/store/8l79hrfgs3sqc0x4p544zkxz502wj4y4-python3.7-pyasn1-0.4.4/lib/python3.7/site-packages','/nix/store/j88dn1df0scz96flq1g4wgpphnfbz5a3-python3.7-schema-0.6.8/lib/python3.7/site-packages','/nix/store/qnzpg1iqbwrjhpjc6h99w328gz31lfv9-python3.7-jsonpath-rw-1.4.0/lib/python3.7/site-packages','/nix/store/mk0xch65ygwqq42g5162mncpmnjbgi8r-python3.7-requests-2.20.1/lib/python3.7/site-packages','/nix/store/zils6n7ql04i5njw01y5xlm5b8jvm0hv-python3.7-urllib3-1.24.1/lib/python3.7/site-packages','/nix/store/rv73j22qykp80w1h1b1v0nshwj1fbfsy-python3.7-idna-2.7/lib/python3.7/site-packages','/nix/store/6k3a8bq435wwp4sn6xk1qz205qzxa19l-python3.7-asn1crypto-0.24.0/lib/python3.7/site-packages','/nix/store/fla3svslcmnir7yr8w3is7img0kxdq72-python3.7-packaging-18.0/lib/python3.7/site-packages','/nix/store/2fy8lhi7hryajhh2p527yqh8r51q40ik-python3.7-pyparsing-2.3.0/lib/python3.7/site-packages','/nix/store/wfl54yjr4gwa6y07r5g4sp5l0acz6ds6-python3.7-pycparser-2.19/lib/python3.7/site-packages','/nix/store/6gfsg5k2lw4v513gynbnfaz12yx15ams-python3.7-cffi-1.11.5/lib/python3.7/site-packages','/nix/store/zp061rvpg2faa81k1rnflxmm1d1li4zi-python3.7-cryptography-2.3.1/lib/python3.7/site-packages','/nix/store/sxwiv6yf0ni6wa9y2j5hwsr8dg3yd9y6-python3.7-pyOpenSSL-18.0.0/lib/python3.7/site-packages','/nix/store/83qzrsx5zwym562bscwdgdry47qgpb60-python3.7-certifi-2018.11.29/lib/python3.7/site-packages','/nix/store/grqc8i2sqzmzkzi3dskik91sp4xi3pwy-python3.7-pysocks-1.6.8/lib/python3.7/site-packages','/nix/store/b8hrpwr27x82zqyvch81zrdv8i2igdxm-python3.7-chardet-3.0.4/lib/python3.7/site-packages','/nix/store/mv181w2mrlwy0y68nb0mbv2c8lckw7w1-python3.7-grandalf-0.6/lib/python3.7/site-packages','/nix/store/fqqk8a885y2rb6x386m8rdw2kc0vbapy-python3.7-asciimatics-1.10.0/lib/python3.7/site-packages','/nix/store/4wrg4g7blmb4dd8f9vbg9c5qp4r464bp-python3.7-pyfiglet-0.8.post0/lib/python3.7/site-packages','/nix/store/0s05jsiinkcjf7chyjmqzpnndpnaf4sz-python3.7-Pillow-5.3.0/lib/python3.7/site-packages','/nix/store/1iyg4kndi8yp62is4xh5zmrvcgcp61sl-python3.7-olefile-0.46/lib/python3.7/site-packages','/nix/store/klqxj9aqzwm0yc5wjyqh0hmw94k30av2-python3.7-wcwidth-0.1.7/lib/python3.7/site-packages','/nix/store/2hzvyknrhys2yl6bw4791w3pfml659fl-python3.7-distro-1.3.0/lib/python3.7/site-packages','/nix/store/jkbn8sak87m67p20z58l19qy2r5mxmg3-python3.7-appdirs-1.4.3/lib/python3.7/site-packages'], site._init_pathinfo()); import re import sys from dvc.main import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main()) ```

As you can see the PYTHONPATH required is actually filled by the wrapped script.

I think the only proper way to solve is to not call dvc daemon via the library module method basically using python -m dvc daemon -q. But instead just calling dvc daemon normally using the absolute path to the dvc script. However the dvc script is generated by setup.py, and not a fixed part of the source code. I have usually used the scripts=['bin/something'] in my setup.py instead of the entry_points configuration which allows this to easily work (https://github.com/iterative/dvc/blob/master/setup.py#L73-L75).

So right now I need to ask around about this problem. https://github.com/NixOS/nixpkgs/pull/54530#issuecomment-458368361

[efiop]

Hi @CMCDragonkai !

How is it going? :slightly_smiling_face: I see you've introduced a patch that is simply calling dvc daemon directly, which seems to fix the issue for you. It is hard for me to wrap my head around what NixOS packages are doing with that substitution that was suggested to you in https://github.com/NixOS/nixpkgs/pull/54530#discussion_r252581188 . I see that it has been quite a while since your last comment in that discussion, did you decide to use pip instead? Or do you plan to get back to packaging it at some point?

Thanks, Ruslan

[efiop]

@CMCDragonkai ? Closing for now as inactive.

[CMCDragonkai]

The MR is ready to be merged https://github.com/NixOS/nixpkgs/pull/54530 with some help, the patch is ready to be used.

Although in the future, I would be more careful with self-recursive programs with it shelling it out to itself. The path should be ensured to work in all situations.

[efiop]

@CMCDragonkai Great news! But I have to point out that 0.24.1 is extremely old now, we've moved a lot since then, and you might have less or more issues now. 0.24.1 is something we would not recommend to be used by anyone these days.

[CMCDragonkai]

This is a good starting point for the next guy to upgrade it!

On 29 October 2019 01:35:06 GMT+11:00, Ruslan Kuprieiev notifications@github.com wrote:

@CMCDragonkai Great news! But I have to point out that 0.24.1 is extremely old now, we've moved a lot since then, and you might have less or more issues now. 0.24.1 is something we would not recommend to be used by anyone these days.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/iterative/dvc/issues/1509#issuecomment-546974686

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.