search

cachix / devenv

Fast, Declarative, Reproducible, and Composable Developer Environments
https://devenv.sh
Apache License 2.0
3.56k stars 259 forks source link

1.0: Rewrite in Python #745

Closed domenkozar closed 2 months ago

domenkozar commented 10 months ago

devenv started as a bash wrapped in Nix to explore the design space.

As a result, there were many subtle bugs and lack of features that were hard to implement.

Previous devenv was 240M in closure and now it's down to 126M.

TODO

Fixes

Fixes https://github.com/cachix/devenv/issues/744 Fixes #693 Fixes #658 Fixes #244 Fixes #436 Fixes #629 Fixes #731 Fixes #83 Fixes #80 Fixes #749 Fixes #507 Fixes #258 Fixes #486 Fixes #745 Fixes #599 Fixes #607

Try it out 🚀

Add devenv.yaml:

inputs:
  devenv:
     url: github:cachix/devenv/python-rewrite

Plain Nix

cachix use devenv
nix-env -if https://install.devenv.sh/python-rewrite

Flakes

nix profile install --accept-flake-config tarball+https://install.devenv.sh/python-rewrite
cloudflare-pages[bot] commented 10 months ago

Deploying with  Cloudflare Pages  Cloudflare Pages

Latest commit: 2ee4450
Status: âś…  Deploy successful!
Preview URL: https://2ea85a40.devenv.pages.dev
Branch Preview URL: https://python-rewrite.devenv.pages.dev

View logs

domenkozar commented 10 months ago

I noticed that --debugger doesn't really work as we need to properly attach stdin.

domenkozar commented 10 months ago

It seems like we're hitting something similar to https://github.com/nix-community/poetry2nix/issues/596, investigating.

fzakaria commented 9 months ago

Failed for me:

❯ devenv shell
• Building shell ...
error:
       at «github:NixOS/nixpkgs/1e383aada51b416c6c27d4884d2e258df201bc11»/lib/modules.nix:346:9:

          345|         options = checked options;
          346|         config = checked (removeAttrs config [ "_module" ]);
             |         ^
          347|         _module = checked (config._module);

       … while evaluating the attribute 'config'

       error: The option `devenv.root' does not exist. Definition values:
       - In `<unknown-file>': "/usr/local/google/home/fmzakari/code/github.com/fzakaria/sqlelf"

âś– Following command exited with code 1:

  /nix/store/y7jg84kb2kr0rln9a33jdlk6vkzkh9ii-nix-2.12.0pre20230216_7c91803/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --system x86_64-linux print-dev-env --profile '/usr/local/google/home/fmzakari/code/github.com/fzakaria/sqlelf/.devenv/gc/shell' 
âś– Building shell in 0.6s.

(i uninstalled devenv old, installed new one -> rentered shell)

domenkozar commented 9 months ago

You'd also need to add to devenv.yaml:

inputs:
  devenv:
    url: github:cachix/devenv/python-rewrite
shyim commented 9 months ago

I tried also today out, but the python used by devenv just does not compile. It hangs with configure and does not proceed

takeda commented 9 months ago

It seems like we're hitting something similar to nix-community/poetry2nix#596, investigating.

@domenkozar, did you figure out how to get around it, or is that bug still present? I'm experiencing this issue once again.

domenkozar commented 9 months ago

It seems like we're hitting something similar to nix-community/poetry2nix#596, investigating.

@domenkozar, did you figure out how to get around it, or is that bug still present? I'm experiencing this issue once again.

See how I override the python in package.nix - that works well!

domenkozar commented 9 months ago

I tried also today out, but the python used by devenv just does not compile. It hangs with configure and does not proceed

What platform? How comes you don't get it from cachix?

volker-schukai commented 9 months ago

I'll comment here once, if there is a better place, please note.

I use the following which works under the 0.63. but not with the 1:

  version:
    url: git+https://gitlab.example.com/myflexpgm.git
    flake: true

  packages = [
    (inputs.myflexpgm.defaultPackage."${builtins.currentSystem}")

-> error: attribute 'currentSystem' missing

domenkozar commented 9 months ago

I'll comment here once, if there is a better place, please note.

I use the following which works under the 0.63. but not with the 1:

  version:
    url: git+https://gitlab.example.com/myflexpgm.git
    flake: true

  packages = [
    (inputs.myflexpgm.defaultPackage."${builtins.currentSystem}")

-> error: attribute 'currentSystem' missing

We've fixed #658 so now you need to use pkgs.stdenv.system.

anoadragon453 commented 9 months ago

Is this branch compatible with being embedded in a flake currently?

domenkozar commented 9 months ago

It's not yet compatible, bumping Nix and flake compatability is what's left to do.

Atry commented 9 months ago

How would this PR affect Flake based configuration? How could the generated flake be located if they are not submitted to git?

shyim commented 9 months ago

The process manager is completely broken for me. It's stopping no processes on macOS. Just enabled MySQL

domenkozar commented 8 months ago

The process manager is completely broken for me. It's stopping no processes on macOS. Just enabled MySQL

The default process manager? Could you provide some logs?

fullykubed commented 8 months ago

It appears that both devenv.local.nix and the dotenv integrations are broken.

No values appear to be loaded from the devenv.local.nix.

While dotenv is able to be enabled, it prints an error message that it cannot find the .env file regardless of whether using the default .env path or even a hardcoded absolute path as a configuration value.

domenkozar commented 8 months ago

It appears that both devenv.local.nix and the dotenv integrations are broken.

No values appear to be loaded from the devenv.local.nix.

While dotenv is able to be enabled, it prints an error message that it cannot find the .env file regardless of whether using the default .env path or even a hardcoded absolute path as a configuration value.

Good point, thanks for catching that! Will make the fixes and make sure the tests cover it.

domenkozar commented 8 months ago

@jclangst it should be fixed, could you give it a try?

fullykubed commented 8 months ago

Tested and works!

fullykubed commented 8 months ago

Found an additional issue when testing this.

Using this version of devenv, the resulting shell has PYTHONPATH set.

That breaks some packages. A good example is pgadmin4-desktopmode which when run results in the following:

Traceback (most recent call last):
  File "/nix/store/7k0g5s570fx4gmzp934xjm23jkkd1jcg-pgadmin-7.0/bin/.pgadmin4-wrapped", line 6, in <module>
    from pgadmin4.pgAdmin4 import main
  File "/nix/store/7k0g5s570fx4gmzp934xjm23jkkd1jcg-pgadmin-7.0/lib/python3.10/site-packages/pgadmin4/pgAdmin4.py", line 102, in <module>
    app = create_app()
  File "/nix/store/7k0g5s570fx4gmzp934xjm23jkkd1jcg-pgadmin-7.0/lib/python3.10/site-packages/pgadmin4/pgadmin/__init__.py", line 759, in create_app
    app.register_blueprint(module)
  File "/nix/store/fizfk41ad8k4qk9wmsi19pnsp5d4ifm5-python3.10-flask-2.2.5/lib/python3.10/site-packages/flask/scaffold.py", line 50, in wrapper_func
    return f(self, *args, **kwargs)
  File "/nix/store/fizfk41ad8k4qk9wmsi19pnsp5d4ifm5-python3.10-flask-2.2.5/lib/python3.10/site-packages/flask/app.py", line 1299, in register_blueprint
    blueprint.register(self, options)
  File "/nix/store/7k0g5s570fx4gmzp934xjm23jkkd1jcg-pgadmin-7.0/lib/python3.10/site-packages/pgadmin4/pgadmin/misc/__init__.py", line 109, in register
    from .cloud import blueprint as module
  File "/nix/store/7k0g5s570fx4gmzp934xjm23jkkd1jcg-pgadmin-7.0/lib/python3.10/site-packages/pgadmin4/pgadmin/misc/cloud/__init__.py", line 29, in <module>
    from pgadmin.misc.cloud.azure import deploy_on_azure, clear_azure_session
  File "/nix/store/7k0g5s570fx4gmzp934xjm23jkkd1jcg-pgadmin-7.0/lib/python3.10/site-packages/pgadmin4/pgadmin/misc/cloud/azure/__init__.py", line 32, in <module>
    from azure.mgmt.rdbms.postgresql_flexibleservers.models import \
ImportError: cannot import name 'NameAvailabilityRequest' from 'azure.mgmt.rdbms.postgresql_flexibleservers.models' (/nix/store/yrchq6xk5pxsscizwzi9zh1ax1ysznq1-python3.10-azure-mgmt-rdbms-10.1.0/lib/python3.10/site-packages/azure/mgmt/rdbms/postgresql_flexibleservers/models/__init__.py)

Unsetting PYTHONPATH fixes the issues.

However, I am not sure if PYTHONPATH is intended to be set in resulting shells or what issues may occur from unsetting it.

Our workaround is a wrapper around the packages that break that removes the PYTHONPATH.

bobvanderlinden commented 8 months ago

I do still run something fishy and I'm not sure why yet. Nor whether it is problematic. Thought I'd share already.

In some of the existing projects I run devenv shell and it shows:

...
• Entering shell
path '.../.devenv/gc/shell' does not contain a '.devenv.flake.nix', searching up

This is the result of devenv calling:

nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --option eval-cache false --system x86_64-linux develop '.../.devenv/gc/shell'

It does so because .../.devenv/gc/shell is a symlink to ~/.local/share/.devenv/gc/1694345793165-shell, which does not exist. I verified (using print("exists?", gc_root, Path(gc_root).exists())) right before running nix develop that it indeed does not exist at that time. Any idea / leads?

domenkozar commented 8 months ago

I do still run something fishy and I'm not sure why yet. Nor whether it is problematic. Thought I'd share already.

In some of the existing projects I run devenv shell and it shows:

...
• Entering shell
path '.../.devenv/gc/shell' does not contain a '.devenv.flake.nix', searching up

This is the result of devenv calling:

nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --option eval-cache false --system x86_64-linux develop '.../.devenv/gc/shell'

It does so because .../.devenv/gc/shell is a symlink to ~/.local/share/.devenv/gc/1694345793165-shell, which does not exist. I verified (using print("exists?", gc_root, Path(gc_root).exists())) right before running nix develop that it indeed does not exist at that time. Any idea / leads?

Thanks, I pushed a fix.

bobvanderlinden commented 8 months ago

Confirmed that the warnings doesn't show anymore :+1: Thanks.

The only thing I'm not too sure of is compatiblity between old/new config+CLI.

A new CLI with old devenv.yaml results in an error like:

error:
       error: flake 'git+file://...' does not provide attribute 'packages.x86_64-linux.devenv.cachix', 'legacyPackages.x86_64-linux.devenv.cachix' or 'devenv.cachix'

An old CLI with new devenv.yaml results in an error like:

error:
       … while updating the lock file of flake 'git+file://...'

       error: unsupported input attribute 'treeHash'

To have some kind of a migration path it would be nice if one of the 2 cases would be compatible. That way we can ask developers to upgrade the CLI and then upgrade devenv.yaml or the other way around. It avoids a lock-step migration with multiple people.

domenkozar commented 8 months ago

Confirmed that the warnings doesn't show anymore đź‘Ť Thanks.

The only thing I'm not too sure of is compatiblity between old/new config+CLI.

A new CLI with old devenv.yaml results in an error like:

error:
       error: flake 'git+file://...' does not provide attribute 'packages.x86_64-linux.devenv.cachix', 'legacyPackages.x86_64-linux.devenv.cachix' or 'devenv.cachix'

An old CLI with new devenv.yaml results in an error like:

error:
       … while updating the lock file of flake 'git+file://...'

       error: unsupported input attribute 'treeHash'

To have some kind of a migration path it would be nice if one of the 2 cases would be compatible. That way we can ask developers to upgrade the CLI and then upgrade devenv.yaml or the other way around. It avoids a lock-step migration with multiple people.

The new CLI should only warn if cachix integration is failing, could you post the full log?

For the old CLI I'm afraid we can't do much because the lock file has changed. I'm open to ideas here.

bobvanderlinden commented 8 months ago

~I noticed it only happens when using direnv(?)~ It happens when using direnv as well as running devenv shell. The full output:

``` direnv: loading ~/tmp/devenv-experiment/.envrc direnv: loading https://raw.githubusercontent.com/cachix/devenv/d1f7b48e35e6dee421cfd0f51481d17f77586997/direnvrc (sha256-YBzqskFZxmNb3kYVoKD9ZixoPXJh1C9ZvTLGFRkauZ0=) direnv: using devenv direnv: .envrc changed, reloading error: error: flake 'git+file:///home/bob.vanderlinden/tmp/devenv-experiment' does not provide attribute 'packages.x86_64-linux.devenv.cachix', 'legacyPackages.x86_64-linux.devenv.cachix' or 'devenv.cachix' ✖ Following command exited with code 1: /nix/store/bnbhch04awczibxch0qg4lbc3aksrcvw-nix-devenv-2.18.0/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --option eval-cache false --system x86_64-linux eval .#devenv.cachix --json • Failed to evaluate .#devenv.cachix • Maybe you need to upgrade to devenv 1.0: https://devenv.sh/getting-started/ [2/5 built, 1/30/31 copied (58.4/93.8 MiB), 10.7/15.5 MiB DL] fetching perl-5.38.0 from https://cache.nixdirenv: ([/nix/store/f84679kq9qvfmhzb2la8m4yqqkba95vl-direnv-2.32.3/bin/direnv export fish]) is taking a while to execute. Use CTRL-C to give up. direnv: updated devenv shell cache ✨ devenv 1.0 is newer than devenv input in devenv.lock. Run `devenv update` to sync. direnv: export +C_INCLUDE_PATH +DEVENV_DOTFILE +DEVENV_PROFILE +DEVENV_ROOT +DEVENV_STATE +IN_NIX_SHELL +LD_LIBRARY_PATH +LIBRARY_PATH +PKG_CONFIG_PATH +name ~PATH ~XDG_CONFIG_DIRS ~XDG_DATA_DIRS ```

It does continue, but it is shown as an error.

To reproduce, here are the files: https://gist.github.com/bobvanderlinden/9957bbbca2cbc83f9f5d9443cd020169

EDIT: I made some changes to avoid logging the errors: https://github.com/bobvanderlinden/devenv/compare/42b06d40dc74f7b8783858b574293865a8ee1405..0dc0ed7b7184834a728e8533d46065a6606a373f

The output now looks like:

$ devenv shell
• Building shell ...
âś” Building shell in 1.0s.
• Entering shell
✨ devenv 1.0 is newer than devenv input in devenv.lock. Run `devenv update` to sync.

EDIT2: I noticed some inconsistent styling in the python files. Should it be formatted using a pre-commit? https://github.com/bobvanderlinden/devenv/compare/0dc0ed7b7184834a728e8533d46065a6606a373f..56afd36

badele commented 7 months ago

I have tested, i seem work !

Good job,

imincik commented 7 months ago

This rewrite looks great ! Congratulations ! When we can expect this PR to be all feature complete and we can start testing ?

bobvanderlinden commented 7 months ago

[...] and we can start testing ?

That's already possible. The cli should work with older devenv configurations. You can install it using:

nix profile install github:cachix/devenv/python-rewrite

I've been using this version with my patches mentioned above. Haven't run into issues.

ankhers commented 7 months ago

I have a few projects at $DAYJOB that do not officially support devenv. In order to work, I setup devenv in the project and just ignored the devenv specific files. Today I started trying to work and devenv fails to initialize. I believe it is because the devenv.lock file is ignored by git. But here is the full log when trying to access that directory.

``` direnv: loading ~/path/to/.envrc direnv: loading https://raw.githubusercontent.com/cachix/devenv/d1f7b48e35e6dee421cfd0f51481d17f77586997/direnvrc (sha256-YBzqskFZxmNb3kYVoKD9ZixoPXJh1C9ZvTLGFRkauZ0=) direnv: using devenv direnv: .envrc changed, reloading warning: creating lock file '/path/to/inner_dir/devenv.lock' The following paths are ignored by one of your .gitignore files: inner_dir/devenv.lock hint: Use -f if you really want to add them. hint: Turn this message off by running hint: "git config advice.addIgnoredFile false" error: … while updating the lock file of flake 'git+file:///path/to?dir=inner_dir' error: program 'git' failed with exit code 1 ✖ Following command exited with code 1: /nix/store/rcgrb68rhxd899lqb0pgwjwy1jcwlrg5-nix-devenv-2.18.0/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --option eval-cache false --system x86_64-linux eval .#devenv.cachix --json • Failed to evaluate .#devenv.cachix • Maybe you need to upgrade to devenv 1.0: https://devenv.sh/getting-started/ warning: creating lock file '/path/to/devenv.lock' evaluating derivation 'git+file:///path/to?dir=inner_dir#devSheThe following paths are ignored by one of your .gitignore files: inner_dir/devenv.lock hint: Use -f if you really want to add them. hint: Turn this message off by running hint: "git config advice.addIgnoredFile false" error: … while updating the lock file of flake 'git+file:///path/to?dir=inner_dir' error: program 'git' failed with exit code 1 ✖ Following command exited with code 1: /nix/store/rcgrb68rhxd899lqb0pgwjwy1jcwlrg5-nix-devenv-2.18.0/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --option eval-cache false --system x86_64-linux print-dev-env --profile '/path/to/inner_dir/.devenv/gc/shell' --option extra-trusted-public-keys '' --option extra-substituters '' direnv: loading the environment failed direnv: error exit status 1 ```
imincik commented 7 months ago

So far everything works in my project:

devenv.nix

{pkgs, ...}: {

  packages = [
    pkgs.postgresql
    pkgs.osm2pgsql
    pkgs.qgis
  ];

  enterShell = ''
    echo "shell"
  '';

  services.postgres = {
    enable = true;
    extensions = e: [e.postgis];

    initdbArgs = [
      "--locale=en_US.UTF-8"
      "--encoding=UTF8"
    ];

    settings = {
      shared_buffers = "1GB";
      work_mem = "50MB";
      maintenance_work_mem = "10GB";
      autovacuum_work_mem = "2GB";
      wal_level = "minimal";
      checkpoint_timeout = "60min";
      max_wal_size = "10GB";
      checkpoint_completion_target = 0.9;
      max_wal_senders = 0;
      random_page_cost = 1.0;

      full_page_writes = "off";
      fsync = "off";
    };

    initialDatabases = [
      {
        name = "osm";
        schema = ./osm.sql;
      }
    ];
  };
}

but I always see following error:

error:
       error: flake 'git+file:///home/imincik/Projects/gis/iso-codes/code' does not provide attribute 'packages.x86_64-linux.devenv.cachix', 'legacyPackages.x86_64-linux.devenv.cachix' or 'devenv.cachix'

âś– Following command exited with code 1:

  /nix/store/bnbhch04awczibxch0qg4lbc3aksrcvw-nix-devenv-2.18.0/bin/nix --show-trace --extra-experimental-features nix-command --extra-experimental-features flakes --option warn-dirty false --option eval-cache false --system x86_64-linux eval .#devenv.cachix --json
• Failed to evaluate .#devenv.cachix
• Maybe you need to upgrade to devenv 1.0: https://devenv.sh/getting-started/

Also, it is reloading all the time even without any change

direnv: .envrc changed, reloading
bobvanderlinden commented 7 months ago

@imincik The patches mentioned in https://github.com/cachix/devenv/pull/745#issuecomment-1722191281 should resolve the confusing error output.

imincik commented 7 months ago

@imincik The patches mentioned in #745 (comment) should resolve the confusing error output.

Sorry, I thought that this fix is already included.

domenkozar commented 7 months ago

~I noticed it only happens when using direnv(?)~ It happens when using direnv as well as running devenv shell. The full output:

It does continue, but it is shown as an error.

To reproduce, here are the files: https://gist.github.com/bobvanderlinden/9957bbbca2cbc83f9f5d9443cd020169

EDIT: I made some changes to avoid logging the errors: https://github.com/bobvanderlinden/devenv/compare/42b06d40dc74f7b8783858b574293865a8ee1405..0dc0ed7b7184834a728e8533d46065a6606a373f

The output now looks like:

$ devenv shell
• Building shell ...
âś” Building shell in 1.0s.
• Entering shell
✨ devenv 1.0 is newer than devenv input in devenv.lock. Run `devenv update` to sync.

EDIT2: I noticed some inconsistent styling in the python files. Should it be formatted using a pre-commit? https://github.com/bobvanderlinden/devenv/compare/0dc0ed7b7184834a728e8533d46065a6606a373f..56afd36

Could you make a PR for these two changes? Thanks a lot :heart:

imincik commented 7 months ago

processes command is missing help text when running devenv --help

devenv --help

...
  inputs     Manage inputs in devenv.yaml. See http://devenv.sh/inputs/
  processes
  search     Search packages matching NAME in nixpkgs input.
...
imincik commented 7 months ago

Service (in this case postgresql) is not terminated by devenv processes stop.

Devenv configuration

devenv.yaml

inputs:
  devenv:
    url: github:cachix/devenv/python-rewrite
  nixpkgs:
    url: github:NixOS/nixpkgs/nixpkgs-unstable

devenv.nix

{ pkgs, ... }:

{
  services.postgres.enable = true;
}

Steps to reproduce

devenv processes stop • Stopping processes ...✖ No processes running. ✖ Stopping processes in 0.0s.


* Check if Postgresql is running (it is)

ps aux | grep postgre imincik 3323122 0.0 0.0 222740 3080 ? Ss 20:02 0:00 /nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash /nix/store/wlblzf9qis78xv59rd15k2vrjwwyslam-postgres imincik 3323124 0.0 0.1 298380 20600 ? S 20:02 0:00 /nix/store/ki3srrjjzqalvh0hd9lmqavp5v9wr9jp-postgresql-14.9/bin/postgres imincik 3323128 0.0 0.0 298380 3852 ? Ss 20:02 0:00 postgres: checkpointer imincik 3323129 0.0 0.0 298380 5628 ? Ss 20:02 0:00 postgres: background writer imincik 3323130 0.0 0.0 298380 5496 ? Ss 20:02 0:00 postgres: walwriter imincik 3323131 0.0 0.0 298944 6828 ? Ss 20:02 0:00 postgres: autovacuum launcher imincik 3323132 0.0 0.0 282948 4240 ? Ss 20:02 0:00 postgres: stats collector imincik 3323133 0.0 0.0 298808 6688 ? Ss 20:02 0:00 postgres: logical replication launcher



NOTE: what was killed by `devenv processes stop` was `.devenv/processes` script.
imincik commented 7 months ago

Search is crashing when running devenv search postgres.

Devenv configuration: same as in https://github.com/cachix/devenv/pull/745#issuecomment-1758211464

Steps to reproduce

• Using Cachix: devenv trace: warning: The type types.string is deprecated. See https://github.com/NixOS/nixpkgs/pull/66346 for better alternative types. Traceback (most recent call last): File "/nix/store/gi2g45cnxf9hk39cgdzr7ycdpk08yjji-python3.11-devenv-1.0/bin/.devenv-wrapped", line 9, in sys.exit(cli()) ^^^^^ File "/nix/store/q85kg00zk53a2i0355v0qczhvid3nq92-python3.11-click-8.1.6/lib/python3.11/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/q85kg00zk53a2i0355v0qczhvid3nq92-python3.11-click-8.1.6/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/nix/store/q85kg00zk53a2i0355v0qczhvid3nq92-python3.11-click-8.1.6/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/q85kg00zk53a2i0355v0qczhvid3nq92-python3.11-click-8.1.6/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/q85kg00zk53a2i0355v0qczhvid3nq92-python3.11-click-8.1.6/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/q85kg00zk53a2i0355v0qczhvid3nq92-python3.11-click-8.1.6/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nix/store/gi2g45cnxf9hk39cgdzr7ycdpk08yjji-python3.11-devenv-1.0/lib/python3.11/site-packages/devenv/cli.py", line 428, in search (key, value["type"], value["default"], value["description"][:80])


KeyError: 'default'
```

Also, my other observation is that the search looks slower than in current stable version. It evaluates even when running the same search command again.

# Steps to reproduce

* Search for hello package (run it twice)
```
time devenv search hello
...
devenv search hello  14.52s user 1.91s system 97% cpu 16.768 total

time devenv search hello
...
devenv search hello  14.71s user 1.99s system 90% cpu 18.436 total
```
imincik commented 7 months ago

Another very important thing for my use cases is ability to follow nixpkgs version from other input flake.

For example, use nixpkgs version from myflake:

inputs:
  myflake:
    url: github:myrepo/myflake

  nixpkgs:
    follows: myflake/nixpkgs

This configuration is currently not allowed by YAML schema validation specified in yaml.py.

I made it working using following patch, but it would be better if url and follows would be mutually exclusive required objects:

diff --git a/src/devenv/yaml.py b/src/devenv/yaml.py
index 825dbcc..9b696f1 100644
--- a/src/devenv/yaml.py
+++ b/src/devenv/yaml.py
@@ -11,7 +11,9 @@ inputsSchema = EmptyDict() | MapPattern(
     Str(),
     Map(
         {
-            "url": Str(),
+            # "url": Str(),
+            Optional("url", default=None): Str(),
+            Optional("follows", default=None): Str(),
             Optional("flake", default=None): Bool(),
             Optional("inputs", default=None): Any(),
             Optional("overlays", default=None): Seq(Str()),
@@ -50,7 +52,7 @@ def validate_and_parse_yaml(dot_devenv_root):

     inputs = {}
     for input, attrs in devenv.get("inputs", {}).items():
-        inputs[input] = {k: attrs[k] for k in ("url", "inputs", "flake") if k in attrs}
+        inputs[input] = {k: attrs[k] for k in ("url", "follows", "inputs", "flake") if k in attrs}

     with open(os.path.join(dot_devenv_root, "flake.json"), "w") as f:
         f.write(json.dumps(inputs))
Nebucatnetzer commented 7 months ago

I haven’t used devenv much but I’m currently testing it out a bit. First I integrated version 0.63 into my project and then migrated it to version 1.0. I didn’t really encounter any issues, however I’m using the declarative Flakes approach so some things like devenv up -d aren’t working/available. Probably because of the task “ Fix flake.nix support”?

The only thing that I missed compared to my personal setup is random ports. I’m investing this as part of my work and one of the soft requirements was that it would be nice to launch multiple instances without port collisions. I was just using this code to generate the port numbers export WEBPORT=$(($RANDOM + 1100)). There’s of course still a chance of getting a collision but they are fairly rare and you can just reload the environment to “fix” it. Not exactly what I would call a pretty solution but at least one doesn’t have to edit a file.

imincik commented 6 months ago

Command devenv test is documented in the list of commands, but it is not clear how to configure it. URL http://devenv.sh/tests/ doesn't exist.

$ devenv
Usage: devenv [OPTIONS] COMMAND [ARGS]...

  https://devenv.sh: Fast, Declarative, Reproducible, and Composable
  Developer Environments.

Options:
  -v, --verbose              Enable verbose output.
  -n, --nix-flags NIX-FLAGS  Flags to pass to Nix. See `man nix.conf 5`.
                             Example: --nix-flags "--option bash-prompt >"
  -d, --debugger             Enable Nix debugger.
  -s, --system TEXT          Nix system to use.
  -o, --offline              Disable network access.
  --help                     Show this message and exit.

Commands:
...
  test       Run tests. See http://devenv.sh/tests/
...
ankhers commented 6 months ago

Command devenv test is documented in the list of commands, but it is not clear how to configure it. URL http://devenv.sh/tests/ doesn't exist.

You can find the documentation at https://python-rewrite.devenv.pages.dev/tests/

imincik commented 6 months ago

Github CI is crashing on out of memory on any attempt to enter the shell. The same project works with current stable version.

Run devenv shell pre-commit run --all --show-diff-on-failure
• Building shell ...
error: out of memory

âś– Building shell in 88.3s.
Error: Process completed with exit code 1.
imincik commented 6 months ago

Command devenv test is documented in the list of commands, but it is not clear how to configure it. URL http://devenv.sh/tests/ doesn't exist.

You can find the documentation at https://python-rewrite.devenv.pages.dev/tests/

Thank you for explanation @ankhers . So it is something like NixOS test for devenv environment.

imincik commented 6 months ago

Python interpreter appears in shell environment even if not configured by devenv.nix.

Devenv configuration

devenv.yaml

inputs:
  devenv:
    url: github:cachix/devenv/python-rewrite
  nixpkgs:
    url: github:NixOS/nixpkgs/nixpkgs-unstable

devenv.nix

{ pkgs, ... }:

{
  services.postgres.enable = true;
}

Steps to reproduce

Python 3.11.5

$ echo $PYTHONPATH /nix/store/h7kkkiyxrl535ik4ivbbbnn9x2fgma7g-honcho-1.1.0/lib/python3.11/site-packages:/nix/store/c2g2jfc4bxk447qh1ddljdw17nwmn4kg-python3.11-setuptools-68.2.2/lib/python3.11/site-packages:/nix/store/ffll6glz3gwx342z0ch8wx30p5cnqz1z-python3-3.11.5/lib/python3.11/site-packages



Is this related to your [discourse comment](https://discourse.nixos.org/t/devenv-pre-1-0-call-for-testing/34050?u=imincik) saying 

"""
Python native system libraries is currently broken, but once we fix this we believe it will work much better than it used to.
"""
imincik commented 6 months ago

When running nix flake show using Devenv's provided nix build it fails.

arathunku commented 5 months ago

:wave: another bug report

Cannot use a python(?) package

All defaults after init

{pkgs, ...}: {
  # https://devenv.sh/basics/
  env.GREET = "devenv";

  packages = [pkgs.awscli2]
}

Package exists

$ devenv search awscli2
• Using Cachix: devenv
trace: warning: The type `types.string` is deprecated. See https://github.com/NixOS/nixpkgs/pull/66346 for better alternative types.
trace: warning: libsForQt5 now uses makeScopeWithSplicing which does not have "overrideScope'", use "overrideScope".
+--------------+---------+------------------------------------------+
| Package      | Version | Description                              |
+--------------+---------+------------------------------------------+
| pkgs.awscli2 | 2.13.33 | Unified tool to manage your AWS services |
+--------------+---------+------------------------------------------+

Error log on shell activiation:

```nix error: … while calling the 'derivationStrict' builtin at :9:12: 8| 9| strict = derivationStrict drvAttrs; | ^ 10| … while evaluating derivation 'devenv-shell' whose name attribute is located at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:348:7 … while evaluating attribute 'DEVENV_PROFILE' of derivation 'devenv-shell' at «none»:0: (source not available) … while evaluating derivation 'devenv-profile' whose name attribute is located at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:348:7 … while evaluating attribute 'passAsFile' of derivation 'devenv-profile' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/build-support/trivial-builders/default.nix:88:7: 87| inherit buildCommand name; 88| passAsFile = [ "buildCommand" ] | ^ 89| ++ (derivationArgs.passAsFile or []); … while evaluating derivation 'awscli2-2.13.33' whose name attribute is located at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:348:7 … while evaluating attribute 'nativeBuildInputs' of derivation 'awscli2-2.13.33' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:392:7: 391| depsBuildBuild = elemAt (elemAt dependencies 0) 0; 392| nativeBuildInputs = elemAt (elemAt dependencies 0) 1; | ^ 393| depsBuildTarget = elemAt (elemAt dependencies 0) 2; … while evaluating derivation 'python3.11-jsonschema-4.19.0' whose name attribute is located at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:348:7 … while evaluating attribute 'nativeBuildInputs' of derivation 'python3.11-jsonschema-4.19.0' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:392:7: 391| depsBuildBuild = elemAt (elemAt dependencies 0) 0; 392| nativeBuildInputs = elemAt (elemAt dependencies 0) 1; | ^ 393| depsBuildTarget = elemAt (elemAt dependencies 0) 2; … while evaluating derivation 'python3.11-pip-23.2.1' whose name attribute is located at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:348:7 … while evaluating attribute 'nativeBuildInputs' of derivation 'python3.11-pip-23.2.1' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:392:7: 391| depsBuildBuild = elemAt (elemAt dependencies 0) 0; 392| nativeBuildInputs = elemAt (elemAt dependencies 0) 1; | ^ 393| depsBuildTarget = elemAt (elemAt dependencies 0) 2; … while evaluating derivation 'python3.11-sphinx-7.2.6' whose name attribute is located at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:348:7 … while evaluating attribute 'propagatedBuildInputs' of derivation 'python3.11-sphinx-7.2.6' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:402:7: 401| depsHostHostPropagated = elemAt (elemAt propagatedDependencies 1) 0; 402| propagatedBuildInputs = elemAt (elemAt propagatedDependencies 1) 1; | ^ 403| depsTargetTargetPropagated = elemAt (elemAt propagatedDependencies 2) 0; … while evaluating derivation 'python3.11-requests-2.31.0' whose name attribute is located at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:348:7 … while evaluating attribute 'propagatedBuildInputs' of derivation 'python3.11-requests-2.31.0' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:402:7: 401| depsHostHostPropagated = elemAt (elemAt propagatedDependencies 1) 0; 402| propagatedBuildInputs = elemAt (elemAt propagatedDependencies 1) 1; | ^ 403| depsTargetTargetPropagated = elemAt (elemAt propagatedDependencies 2) 0; … while calling 'getOutput' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/attrsets.nix:962:23: 961| */ 962| getOutput = output: pkg: | ^ 963| if ! pkg ? outputSpecified || ! pkg.outputSpecified … while calling anonymous lambda at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:308:13: 307| (map (drv: drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHostPropagated" depsHostHostPropagated)) 308| (map (drv: drv.__spliced.hostTarget or drv) (checkDependencyList "propagatedBuildInputs" propagatedBuildInputs)) | ^ 309| ] … while calling anonymous lambda at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/lists.nix:164:29: 163| */ 164| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list); | ^ 165| … from call site at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/lists.nix:164:32: 163| */ 164| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list); | ^ 165| … while calling anonymous lambda at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:261:73: 260| checkDependencyList = checkDependencyList' []; 261| checkDependencyList' = positions: name: deps: flip imap1 deps (index: dep: | ^ 262| if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep … from call site at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/stdenv/generic/make-derivation.nix:262:8: 261| checkDependencyList' = positions: name: deps: flip imap1 deps (index: dep: 262| if isDerivation dep || dep == null || builtins.isString dep || builtins.isPath dep then dep | ^ 263| else if isList dep then checkDependencyList' ([index] ++ positions) name dep … while calling 'isDerivation' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/attrsets.nix:670:5: 669| # Value to check. 670| value: value.type or null == "derivation"; | ^ 671| … while calling 'checkDrv' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/development/interpreters/python/mk-python-derivation.nix:174:16: 173| 174| checkDrv = drv: | ^ 175| if (isPythonModule drv) && (isMismatchedPython drv) … from call site at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/development/interpreters/python/mk-python-derivation.nix:175:11: 174| checkDrv = drv: 175| if (isPythonModule drv) && (isMismatchedPython drv) | ^ 176| then throwMismatch drv … while calling 'isPythonModule' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/development/interpreters/python/mk-python-derivation.nix:129:22: 128| validatePythonMatches = attrName: let 129| isPythonModule = drv: | ^ 130| # all pythonModules have the pythonModule attribute … from call site at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/tools/admin/awscli2/default.nix:22:17: 21| }); 22| urllib3 = prev.urllib3.overridePythonAttrs (prev: { | ^ 23| format = "setuptools"; … while calling 'overridePythonAttrs' at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/development/interpreters/python/python-packages-base.nix:21:35: 20| passthru = (previousAttrs.passthru or { }) // { 21| overridePythonAttrs = newArgs: makeOverridablePythonPackage f (overrideWith newArgs); | ^ 22| }; … from call site at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/trivial.nix:430:7: 429| { # TODO: Should we add call-time "type" checking like built in? 430| __functor = self: f; | ^ 431| __functionArgs = args; … while calling anonymous lambda at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/development/interpreters/python/python-packages-base.nix:16:63: 15| # This function introduces `overridePythonAttrs` and it overrides the call to `buildPythonPackage`. 16| makeOverridablePythonPackage = f: lib.mirrorFunctionArgs f (origArgs: | ^ 17| let … from call site at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/trivial.nix:430:7: 429| { # TODO: Should we add call-time "type" checking like built in? 430| __functor = self: f; | ^ 431| __functionArgs = args; … while calling anonymous lambda at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/customisation.nix:84:17: 83| in 84| mirrorArgs (origArgs: | ^ 85| let … from call site at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/lib/trivial.nix:430:7: 429| { # TODO: Should we add call-time "type" checking like built in? 430| __functor = self: f; | ^ 431| __functionArgs = args; … while calling anonymous lambda at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/development/interpreters/python/mk-python-derivation.nix:30:1: 29| 30| { name ? "${attrs.pname}-${attrs.version}" | ^ 31| at «github:NixOS/nixpkgs/51a01a7e5515b469886c120e38db325c96694c2f»/pkgs/development/interpreters/python/mk-python-derivation.nix:110:1: 109| 110| assert (pyproject != null) -> (format == null); | ^ 111| ```

awscli2 works without devenv:

$ nix-shell -p awscli2 --command "aws --version"
aws-cli/2.13.33 Python/3.11.5 Linux/6.2.0-36-generic source/x86_64.ubuntu.22 prompt/off
imincik commented 5 months ago

Devenv is failing on versionGLIBC_2.38' not found`.

devenv.nix

{ pkgs, ... }: {

  packages = [
    # qgis currently works only on linux
  ] ++ pkgs.lib.optionals pkgs.stdenv.isLinux [
    pkgs.qgis
    pkgs.grass
  ];

  languages.python = {
    enable = true;
    package = pkgs.python3.withPackages (p: [
      p.rasterio
      p.fiona
    ]);
  };

  services = {
    postgres = {
      enable = true;
      extensions = e: [ e.postgis ];
    };
  };
}

Command:

devenv up

/nix/store/cl6bxs1bad3bbw6fhhrvdq1vlihqjqqg-python3-3.11.4/bin/python3.11: /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libm.so.6: version `GLIBC_2.38' not found (required by /nix/store/jippvc3nhbzl961dfs6xxr1n4hiiknhh-devenv-profile/lib/libpython3.11.so.1.0)
/nix/store/cl6bxs1bad3bbw6fhhrvdq1vlihqjqqg-python3-3.11.4/bin/python3.11: /nix/store/vq3sdi8l15rzfl5zvmwpafrzis4sm6xf-glibc-2.37-8/lib/libc.so.6: version `GLIBC_2.38' not found (required by /nix/store/jippvc3nhbzl961dfs6xxr1n4hiiknhh-devenv-profile/lib/libpython3.11.so.1.0)

UPDATE: I was missing github:cachix/devenv/python-rewrite input in devenv.yaml. When added, this error doesn't show up anymore.

imincik commented 5 months ago

Can't use Python configured via languages.python.

Devenv configuration

devenv.yaml

inputs:
  devenv:
    url: github:cachix/devenv/python-rewrite
  nixpkgs:
    url: github:NixOS/nixpkgs/nixpkgs-unstable

devenv.nix

{ pkgs, ... }:

{
  languages.python = {
    enable = true;
    package = pkgs.python3.withPackages (p: [
      p.rasterio
      p.fiona
    ]);
  };
}

Steps to reproduce

misumisumi commented 5 months ago

I'm sorry if this has already been tracked, but with the current method of declaring LD_LIBRARY_PATH inside the wrapper, when managing a project using poetry, it is difficult to access the shared library because poetry does not use wrapped python. This will cause problems.

misumisumi commented 5 months ago

image For some reason it seems like the interpreter before wrapping is being referenced