It is possible to add a dependency of a private git repo?

Hammond95 commented 5 years ago

[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have searched the documentation and believe that my question is not covered.

Question

I've just stated to use poetry, I quite like some of the features, but i'm now stuck since installing a private repo doesn't seem to work...

toml file extract

[tool.poetry.dependencies]
python = "~3.6"  # Compatible python versions must be declared here
toml = "^0.9"

# Git dependencies
myprivaterepo = { git = "git@github.com:myorganization/myprivaterepo.git", branch = "master" }

I've also tried with different git url forms:

https://github.com/myorganization/myprivaterepo.git
git+ssh://git@github.com/myorganization/myprivaterepo.git

the corresponding pip command i have used till now is:

pip install git+ssh://git@github.com/myorganization/myprivaterepo.git

This is the error i get:

Updating dependencies
Resolving dependencies... (0.0s)

[AssertionError]  

Exception trace:
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 /Users/mdeluca/.poetry/lib/poetry/console/application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 /Users/mdeluca/.poetry/lib/poetry/console/commands/command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 /Users/mdeluca/.poetry/lib/poetry/_vendor/py3.6/cleo/commands/command.py in execute() at line 107
   return self.handle()
 /Users/mdeluca/.poetry/lib/poetry/console/commands/update.py in handle() at line 41
   return installer.run()
 /Users/mdeluca/.poetry/lib/poetry/installation/installer.py in run() at line 76
   self._do_install(local_repo)
 /Users/mdeluca/.poetry/lib/poetry/installation/installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 /Users/mdeluca/.poetry/lib/poetry/puzzle/solver.py in solve() at line 38
   packages, depths = self._solve(use_latest=use_latest)
 /Users/mdeluca/.poetry/lib/poetry/puzzle/solver.py in _solve() at line 171
   self._package, self._provider, locked=locked, use_latest=use_latest
 /Users/mdeluca/.poetry/lib/poetry/mixology/__init__.py in resolve_version() at line 7
   return solver.solve()
 /Users/mdeluca/.poetry/lib/poetry/mixology/version_solver.py in solve() at line 79
   next = self._choose_package_version()
 /Users/mdeluca/.poetry/lib/poetry/mixology/version_solver.py in _choose_package_version() at line 380
   for incompatibility in self._provider.incompatibilities_for(version):
 /Users/mdeluca/.poetry/lib/poetry/puzzle/provider.py in incompatibilities_for() at line 447
   for dep in dependencies
 /Users/mdeluca/.poetry/lib/poetry/puzzle/provider.py in <listcomp>() at line 447
   for dep in dependencies
 /Users/mdeluca/.poetry/lib/poetry/mixology/incompatibility.py in __init__() at line 59
   assert by_ref[ref] is not None

jasongi-actu commented 5 years ago

You need to specify the protocol:

myprivaterepo = { git = "ssh://git@github.com/myorganization/myprivaterepo.git", branch = "master" }

Or just add using the --git flag

poetry add myprivaterepo --git ssh://git@github.com/myorganization/myprivaterepo.git

See here for more info on the add command https://poetry.eustace.io/docs/cli/#add

sdispater commented 5 years ago

@Hammond95 Is your issue fixed now?

Peque commented 5 years ago

@jasongi-actu 's solution did not fix the issue for me. I get this error:

[RuntimeError]
Unable to retrieve the package version for /tmp/pypoetry-git-mypackagekg2xNw

Dependency is declared as:

mypackage = { git = "ssh://git@private.instance.url:12345/organization/mypackage.git", tag = "0.1.2" }

Update

Using Poetry version 1.0.0a2. Tried with branch = "master" instead of tag = "x.x.x", without success.

Peque commented 5 years ago

@jasongi-actu Did you try that command you suggested? Did it work for you? I even tried on a public repository I have in GitHub (only specifying the ssh:// protocol and it did not work. The documentation you linked seems to not mention anything about SSH protocol (assumes HTTPS, with no SSH authentication).

jasongi-actu commented 5 years ago

Yes I use that command all the time in multiple public/private github repos. I think the next step in solving this would be to get a reproducible test case.

Also, are you running PIP <= 18.1? There are issues with git dependencies and poetry using pip 19.xx

Peque commented 5 years ago

@jasongi-actu I am indeed running pip==18.1:

$ poetry run pip --version
pip 18.1 from [...]

I can reproduce the error by trying to install a public repo of my own:

[tool.poetry.dependencies]
markdownreveal = { git = "ssh://git@github.com/Peque/markdownreveal.git", branch = "master" }

Whereas I am able to:

git clone ssh://git@github.com/Peque/markdownreveal.git`

I guess if you fork it and are able to install from your own fork using the SSH protocol, then it is related to differences between our systems/configurations/environments, not depending on the project that it is to be installed. In which case I am not sure how to proceed to debug.

Peque commented 5 years ago

@jasongi-actu Can you reproduce the issue when forking markdownreveal and setting this dependency?

[tool.poetry.dependencies]
markdownreveal = { git = "ssh://git@github.com/jasongi-actu/markdownreveal.git", branch = "master" }

Alternatively, could you provide a repo that works fine for you so that I can fork it and try it myself?

dsevero commented 5 years ago

@Peque I ran poetry add <repo> --git ssh://git@github.com/<org>/<repo>.git -vvv and it worked like a charm!


Poetry
======

 * Version: 0.12.12
 * Python:  3.7.1

Virtualenv
==========

 * Python:         3.7.1
 * Implementation: CPython
 * Path:           /home/severo/.cache/pypoetry/virtualenvs/<repo>-py3.7
 * Valid:          True

System
======

 * Platform: linux
 * OS:       posix
 * Python:   /home/severo/miniconda3/envs/<repo>

pip 19.1 from /home/severo/.cache/pypoetry/virtualenvs/<repo>-py3.7/lib/python3.7/site-packages/pip (python 3.7)

hope it helps!

Hammond95 commented 5 years ago

Sorry for the late reply, tried @jasongi-actu proposed solution (adding the ssh protocol), but this still doesn't work...

This is the error I get:

[CalledProcessError] Command '['git', 'clone', 'ssh://git@github.com:Organization/myRepo.git', '/var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000 gp/T/pypoetry-git-myproject243mqkhk']' returned non-zero exit status 128.

Running manually the corresponding command

git clone ssh://git@github.com:Organization/myRepo.git /var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000gp/T/pypoetry-git-myproject243mqkhk

results into the following error:

Cloning into '/var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000gp/T/pypoetry-git-myproject243mqkhk'... ssh: Could not resolve hostname github.com:Organization: nodename nor servname provided, or not known fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

running manually the same command but without the ssh:// in front

git clone git@github.com:Organization/myRepo.git /var/folders/d8/0yv7fd_j5rn0cjrsk5dp09cc0000gp/T/pypoetry-git-myproject243mqkhk

works fine.

I get the same errors with the poetry add command.

PS. I am using poetry version 0.12.11

Hammond95 commented 5 years ago

Updating poetry to 0.12.14 partially solved the issue, it manage to get to the resolving dependencies phase, but then it fails because it can't find pip.

Installing dependencies from lock file

Package operations: 55 installs, 0 updates, 0 removals

  - Installing pycparser (2.19)

[EnvCommandError]
Command ['/Users/mdeluca/Library/Caches/pypoetry/virtualenvs/mypackage-py3.6/bin/python', '-m', 'pip', 'install', '
--no-deps', 'pycparser==2.19'] errored with the following output:
/Users/mdeluca/Library/Caches/pypoetry/virtualenvs/mypackage-py3.6/bin/python: No module named pip

I didn't created this env (maybe poetry did?); I am using a conda env to make poetry use a python3.6 interpreter.

thedrow commented 5 years ago

There's an infinite loop somewhere. When I run poetry add dependencies --git git@github.com:dry-python/dependencies.git -vvv I get the following repeated log message:

   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5
   1: fact: dependencies (0.16.dev) requires Python ~2.7 || ^3.5

It runs forever and reaches 100% CPU. The repository is public.

Is this the same bug or a different one?

ekhaydarov commented 5 years ago

ummmm the --git option does not exist anymore?

[NoSuchOptionException]
The "--git" option does not exist.

you can just put it directly into toml file and run poetry update

kasteph commented 4 years ago

@Hammond95 I will close this issue now since your particular issue seems to be resolved and @dsevero has also offered an alternative solution. Furthermore, you can add git dependencies since version 1 using just poetry add without any options.

@thedrow that looks like a different bug and the --git option does not exist since version 1.0 as @ekhaydarov has said.

darakian commented 4 years ago

@stephsamson

Can I request a reopen of this issue? I'm using poetry 1.0.0 withpip 19.2.3 and get the

[RuntimeError]
Unable to retrieve the package version for /var/folders/3g/fpzgglls03vbzcvbwvgw0r540000gn/T/pypoetry-git-myrepo-fooGTw158

error.

Digging into the code it looks like lines 340-348 of poetry/puzzle/provider.py are my issue.

                if not result["version"]:
                    # The version could not be determined
                    # so we raise an error since it is mandatory
                    print("Result {}".format(result))
                    raise RuntimeError(
                        "Unable to retrieve the package version for {}".format(
                            directory
                        )
                    )

I added a simple print out of the result dict and get

Result {'install_requires': [], 'extras_require': {}, 'version': None, 'python_requires': None, 'name': u'myrepofoo'}

In my case I'm matching on a tag which is actually how we're versioning our dependency. In our setup.py file we have a more or less static version set, but if there's a method that poetry would accept then I'd happily add that. I've attempted using the branch directive as well and get the same behavior.

Thanks.

finswimmer commented 4 years ago

Hey @darakian ,

please open a new issue, where you describe your problem by showing your pyproject.toml, what steps need to be done to reproduce your problem, what's your expected behavior and what happens instead.

Thanks!

fin swimmer

darakian commented 4 years ago

Hey @finswimmer

I ended up realizing that it was an error on my end (sorry). Turns out that our code was using distutils rather than setuptools and the lines https://github.com/python-poetry/poetry/blob/636ce8b0eba7dfa390b3fd961d1b9fb533d5d033/poetry/puzzle/provider.py#L324-L332 were borking out as the egg_info command was failing. A one line change of

from distutils.core import setup

to

from setuptools import setup

in our setup.py file fixed up the poetry install.

dazza-codes commented 4 years ago

poetry 1.x

[NoSuchOptionException]
The "--git" option does not exist.

Not enough information in the help too:

$ poetry --version
Poetry version 1.0.2

$ poetry help add
USAGE
  poetry add [-D] [-E <...>] [--optional] [--python <...>] [--platform <...>] [--allow-prereleases] [--dry-run] <name1> ... [<nameN>]

ARGUMENTS
  <name>                 The packages to add.

OPTIONS
  -D (--dev)             Add as a development dependency.
  -E (--extras)          Extras to activate for the dependency. (multiple values allowed)
  --optional             Add as an optional dependency.
  --python               Python version for which the dependency must be installed.
  --platform             Platforms for which the dependency must be installed.
  --allow-prereleases    Accept prereleases.
  --dry-run              Output the operations but do not execute anything (implicitly enables --verbose).

GLOBAL OPTIONS
  -h (--help)            Display this help message
  -q (--quiet)           Do not output any message
  -v (--verbose)         Increase the verbosity of messages: "-v" for normal output, "-vv" for more verbose output and "-vvv" for debug
  -V (--version)         Display this application version
  --ansi                 Force ANSI output
  --no-ansi              Disable ANSI output
  -n (--no-interaction)  Do not ask any interactive question

DESCRIPTION
  The add command adds required packages to your pyproject.toml and installs them.

  If you do not specify a version constraint, poetry will choose a suitable one based on the available package versions.

$ poetry add ssh://git@gitlab.com/org/private-repo.git

[ValueError]
Could not parse version constraint: //git@gitlab.com/org/private-repo.git

😞

finswimmer commented 4 years ago

@dazza-codes: The correct syntax would be:

$ poetry add git+ssh://git@gitlab.com/org/private-repo.git

fin swimmer

rickysugiarta commented 3 years ago

can use token as well

$ poetry add git+https://GIT_TOKEN@github.com/org/privaterepo.git#BRANCH

john012343210 commented 3 years ago

can use token as well

$ poetry add git+https://GIT_TOKEN@github.com/org/privaterepo.git#BRANCH

but this might make the whole dependency chain token dependent, if somebody revokes that token, we have to manually change this token, does ssh link sounds better?

venaturum commented 3 years ago

can use token as well

$ poetry add git+https://GIT_TOKEN@github.com/org/privaterepo.git#BRANCH

@InnerTaste what is GIT_TOKEN in this example? An environment variable? Or a value which gets hardcoded into pyproject.toml?

rickysugiarta commented 3 years ago

@InnerTaste what is GIT_TOKEN in this example? An environment variable? Or a value which gets hardcoded into pyproject.toml?

@venaturum refer to this https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token yeah, after running the command the token gets hardcoded into pyproject.toml

can use token as well $ poetry add git+https://GIT_TOKEN@github.com/org/privaterepo.git#BRANCH

but this might make the whole dependency chain token dependent, if somebody revokes that token, we have to manually change this token, does ssh link sounds better?

yeah may have drawbacks too, but this https/token method works in docker because my circumstances need the private repo to be installed inside a fresh container sorry, im new on this poetry/private git things

niderhoff commented 3 years ago

@dazza-codes: The correct syntax would be:
$ poetry add git+ssh://git@gitlab.com/org/private-repo.git
fin swimmer

Thank you, this really did the trick. However, I had to add -v to the command (verbose output) because otherwise it would only ask for my SSH-key password once. But in fact poetry was accessing the repository 3 times and the other password prompts did not appear, so the command got stuck.

kivancyuksel commented 3 years ago

can use token as well

$ poetry add git+https://GIT_TOKEN@github.com/org/privaterepo.git#BRANCH

Do you know how to specify GIT_TOKEN as environment variable, or something similar in pyproject.toml? This works:

[tool.poetry.dependencies]
python = "^3.8"
package_name = {git = "https://<token>@github.com/<user>/<repo>", rev = "0.1.3"}

But I don't want to specify the token directly in pyproject.toml, since it will be tracked by git, and I might decide to make my repo public in the future.

dazza-codes commented 3 years ago

I would not recommend using a hard coded TOKEN unless it can be shared among users of an organization or protected in CI/CD workflows. Using an ssh protocol allows any user or CI/CD workflow with the required ssh credentials to get the library OK. It works well when the ssh credentials are coupled with an ssh-agent.

kivancyuksel commented 3 years ago

I would not recommend using a hard coded TOKEN unless it can be shared among users of an organization or protected in CI/CD workflows. Using an ssh protocol allows any user or CI/CD workflow with the required ssh credentials to get the library OK. It works well when the ssh credentials are coupled with an ssh-agent.

So, is this mean that it is not possible to specify the TOKEN using an environment variable, and the only way I can use it is to hard code it into pyproject.toml ?

niderhoff commented 3 years ago

It means that if you are fortunate enough to work on a *nix system you can use ssh access by key and ads the key to your ash agent after you set it up so it should login automatically. However this is not possible using http and token based authentication.

In our Organisation we actually solved the problem by setting up a private pypi repository where a package is updated whenever we push a tag to the git repository. We were then able to add the private repository to the pip config as "extra index".

kivancyuksel commented 3 years ago

It means that if you are fortunate enough to work on a *nix system you can use ssh access by key and ads the key to your ash agent after you set it up so it should login automatically. However this is not possible using http and token based authentication.

In our Organisation we actually solved the problem by setting up a private pypi repository where a package is updated whenever we push a tag to the git repository. We were then able to add the private repository to the pip config as "extra index".

Thank you for your explanations. However, I still couldn't get an answer to my question. Surely, doing it as you described is a way to do it, and maybe the best way, I don't know. I am just trying to discover my other options, if there are any.

ekhaydarov commented 3 years ago

@sdispater is very interesting character and did not want pyproject.toml to be able to parse environment variable. there is a very strange workaround to use environment variables noted here

niderhoff commented 3 years ago

@sdispater is very interesting character and did not want pyproject.toml to be able to parse environment variable. there is a very strange workaround to use environment variables noted here

The solution described in the issue you linked would work with a pypi repository but not a git repository, correct?

EMCP commented 3 years ago

$ poetry add git+ssh://git@github.com/MYORG/MY-PRIVATE-REPO.git

  TypeError

  expected string or bytes-like object

  at ~/anaconda3/envs/meow/lib/python3.9/site-packages/poetry/core/utils/helpers.py:27 in canonicalize_name
       23│ _canonicalize_regex = re.compile(r"[-_]+")
       24│ 
       25│ 
       26│ def canonicalize_name(name):  # type: (str) -> str
    →  27│     return _canonicalize_regex.sub("-", name).lower()
       28│ 
       29│ 
       30│ def module_name(name):  # type: (str) -> str
       31│     return canonicalize_name(name).replace(".", "_").replace("-", "_")

I can SSH clone this repo from the environment I am using.. please forgive me I am brand new to poetry but.. my usual workflow involves a LOT of pip installing off private git repos and just trying to get started ran into this error

CmdQ commented 3 years ago

I also throw in my two cents. Most of our package come from PyPI, but a few need to be pulled from a private and SSH key-file-only Gerrit repository (no anonymous, no username/password).

po add git+ssh://code.some-domain.de:29418/foo/bar#v2.1.3+poetry -vv
Using virtualenv: /home/cmdq/.cache/pypoetry/virtualenvs/foobar-j0Abac2_-py3.8

  Stack trace:

  11  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py:131 in run
      status_code = command.handle(parsed_args, io)

  10  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:120 in handle
      status_code = self._do_handle(args, io)

   9  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:171 in _do_handle
      return getattr(handler, handler_method)(args, io, self)

   8  ~/.poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py:92 in wrap_handle
      return self.handle()

   7  ~/.poetry/lib/poetry/console/commands/add.py:106 in handle
      requirements = self._determine_requirements(

   6  ~/.poetry/lib/poetry/console/commands/init.py:320 in _determine_requirements
      requires = self._parse_requirements(requires)

   5  ~/.poetry/lib/poetry/console/commands/init.py:410 in _parse_requirements
      package = Provider.get_package_from_vcs(

   4  ~/.poetry/lib/poetry/puzzle/provider.py:193 in get_package_from_vcs
      git.clone(url, tmp_dir)

   3  ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/vcs/git.py:212 in clone
      return self.run("clone", "--recurse-submodules", repository, str(dest))

   2  ~/.poetry/lib/poetry/_vendor/py3.8/poetry/core/vcs/git.py:302 in run
      subprocess.check_output(["git"] + list(args), stderr=subprocess.STDOUT)

   1  ~/.pyenv/versions/3.8.6/lib/python3.8/subprocess.py:411 in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

  CalledProcessError

  Command '['git', 'clone', '--recurse-submodules', 'code.some-domain.de:29418/foo/bar', '/tmp/pypoetry-git-foobarnplduj04']' returned non-zero exit status 128.

  at ~/.poetry/lib/poetry/utils/_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.```

A simple checkout on the console is as easy as git clone "ssh://$USER@code.some-domain.de:29418/foo/bar" once Gerrit has your private key, no problems there.

Analysis

It's no wonder this doesn't work. Looking at the built command line we see

'git'
'clone'
'--recurse-submodules'
'code.some-domain.de:29418/foo/bar'
'/tmp/pypoetry-git-foobarnplduj04'

What's missing is at least the correct user (which I would not want hard-coded in the pyproject.toml BTW) and any authentication.

Already tried

I already tried setting Git's GIT_SSH_COMMAND which is what our CI did before when we used pip. It seems that is not respected though Poetry.
I tried encoding at least the username in the URL, but how would Poetry know to pickup the key file.
Having it already available via ssh-agent that doesn't solve it.
I added a tool.poetry.source but there isn't an option for key files or similar. Besides, they would anyway to be different on CI vs. on the developer's machine.

We cannot upload that proprietary to PyPI and won't be getting an internal package repository soon. Also any other authentication method is out. The later is the core of the problem in our setup IMHO.

Did I overlook a way to set this up? Otherwise please reopen.

venaturum commented 3 years ago

@CmdQ, at my work we publish packages to a private pypi feed in Azure DevOps. Prior to this though we would just publish to a simple pypi server (https://pypi.org/project/pypiserver/) that was running on an on-site VM connected to our network. Both of these work with poetry. Not a solution to the issue at hand but perhaps it's an option.

CmdQ commented 3 years ago

Update for devs/affected

Root cause

We found the core problem: the port specification. If we in any way try to inject the port into the URL, the process fails. That's because for poetry add git+ssh://code.some-domain.de:29418/foo/bar Poetry will drop the SSH protocol when forwarding to git: git clone code.some-domain.de:29418/foo/bar

Git—without knowing that SSH is meant—doesn't treat the number as a port but part of the repo path, and that fails. If you on the other hand manually run git clone ssh://code.some-domain.de:29418/foo/bar (note the prefix) it works.

Why isn't the SSH protocol specification forwarded to Git? I'd consider that a bug of Poetry.

Solution

Don't try to add the port via Poetry/command line but set it in your ~/.ssh/config:

Host code.some-domain.de
        User ci_user
        Port 29418

You can/must do this for your CI image, too.
Afterwards poetry add git+ssh://code.some-domain.de:/foo/bar will work.

alexeckert commented 2 years ago

Hi, I think it would still be a valuable feature if one could use env vars or the poetry config methods to set the token for https+git. E.g. it should be simpler to pass a token into Docker and CI/CD than setting up ssh.

davidcortesortuno commented 2 years ago

I'm trying to add a private repo via ssh (I can successfully clone the repo in the machine I'm using poetry)

poetry add "git+ssh://git@github.com/MyOrganization/geometry_tools.git" -v

But I get an error similar to one posted above:

...
1  /opt/***/lib/python3.8/subprocess.py:415 in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

  CalledProcessError

  Command '['git', '--git-dir', '/tmp/pypoetry-git-geometry_tools3mkgvhkq/.git', '--work-tree', '/tmp/pypoetry-git-geometry_tools3mkgvhkq', 'checkout', 'master']' returned non-zero exit status 1.

  at /opt/***/lib/python3.8/site-packages/poetry/utils/_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.

Should I raise a new issue for this? I tried to look into previous issues but none seems to be similar to this. I'm not sure if poetry has problems parsing the ssh specification.

I'm using poetry 1.1.12

systematicguy commented 2 years ago

Guys, I have just spent a tremendous time trying to make this work and my issue was the passphrase for ssh. I am on windows, I solved using the knowledge to be found on the following pieces of wisdom: https://stackoverflow.com/a/68003205/13534526 https://stackoverflow.com/a/53606760/13534526 I also include Start-SshAgent in my powershell profile just to make sure a reboot does not affect me.

jxddk commented 2 years ago

I'm trying to add a private repo via ssh (I can successfully clone the repo in the machine I'm using poetry)

poetry add "git+ssh://git@github.com/MyOrganization/geometry_tools.git" -v

But I get an error similar to one posted above:

...
1  /opt/***/lib/python3.8/subprocess.py:415 in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

  CalledProcessError

  Command '['git', '--git-dir', '/tmp/pypoetry-git-geometry_tools3mkgvhkq/.git', '--work-tree', '/tmp/pypoetry-git-geometry_tools3mkgvhkq', 'checkout', 'master']' returned non-zero exit status 1.

  at /opt/***/lib/python3.8/site-packages/poetry/utils/_compat.py:217 in run
      213│                 process.wait()
      214│                 raise
      215│             retcode = process.poll()
      216│             if check and retcode:
    → 217│                 raise CalledProcessError(
      218│                     retcode, process.args, output=stdout, stderr=stderr
      219│                 )
      220│         finally:
      221│             # None because our context manager __exit__ does not use them.

Should I raise a new issue for this? I tried to look into previous issues but none seems to be similar to this. I'm not sure if poetry has problems parsing the ssh specification.

I'm using poetry 1.1.12

I have the same issue; it appears that the /tmp/ directory does not exist

abn commented 2 years ago

Please raise a new issue with specific details if this is still affecting users.

Note that #5428 should improve git dependency handling considerably. Please try that branch before raising new issues.

fredrikaverpil commented 2 years ago

~~I can't add ssh/git links at all. Moved my comment to https://github.com/python-poetry/poetry/issues/4152#issuecomment-1119706989~~

Works fine now with 1.1.13!

poetry add git+ssh://git@github.com/privateorg/privaterepo.git#main

jchacks commented 2 years ago

By the way poetry defaults to checking out master rather than main (the default for new github repos). Could trip up people using github. This was catching me out and resulting in a CalledProcessError:

CalledProcessError

  Command '['git', '--git-dir', '/tmp/pypoetry-git-packageqxxsy9m5/.git', '--work-tree', '/tmp/pypoetry-git-packageqxxsy9m5', 'checkout', 'master']' returned non-zero exit status 1.

Easy fix is to specify the rev to point to main e.g.: poetry add git+ssh://git@github.com:jchacks/package.git#main or in the pyproject.toml package = {git = "git@github.com:jchacks/package.git", rev="main"}

fredmanre commented 1 year ago

A simple way is doing the following:

on your pyproject.toml:

[tool.poetry.dependencies]
...
myprivaterepo = { git = "ssh://git@github.com/myorganization/myprivaterepo.git", branch = "master" }

if your package name has spaces:

"my privaterepo" = { git = "ssh://git@github.com/myorganization/myprivaterepo.git", branch = "master" }

if you need a particular revision:

"my privaterepo" = { git = "ssh://git@github.com/myorganization/myprivaterepo.git", rev = "0.1.x" }

if you don't want to specify the token for authentication you can use environment variables:

"my repository" = { git = "https://oauth2:$GITLAB_TOKEN@gitlab.com/myorganization/myprivaterepo.git", branch = "master" }

mohitjain2504 commented 1 year ago

Did anyone try installing dependencies that have recursive dependencies provided as Git repos? I have a project where I want to install repo C, which depends on repo B, and a project in repo B is dependent on repo A.

When I try to install project in repo C using the command poetry add git+https://GIT_TOKEN@github.com/org_name/repoC#feat/branch I get the error

Invalid git url "git+https://GIT_TOKEN@github.com/org_name/repoB@master" This is specifically happening because the parent repository are using requirement.txt and pip to install the dependency while my current project uses poetry.

Is there a way to resolve it?

P.S. both the parent repos requires private token for access.

fredrikaverpil commented 1 year ago

@mohitjain2504 yes it will work, but you have to stop using requirements.txt and instead define your dependencies in a pyproject.toml file instead. You can continue using pip to install those projects too: pip install . or pip install -e ..

Note that those projects don't need to adopt poetry. They just need to specify their dependencies like this: https://peps.python.org/pep-0631/

The main difference you'll see about this setup is that your project originally using requirement.txt will now have to be buildable and so you'll also have to specify the project package in the pyproject.toml. This also makes sense as now you want other projects to depend on it. See more here: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/#declaring-project-metadata

Rather than using a "vanilla" pyproject.toml setup, you can also adopt nice tools such as e.g. hatch/hatchling (great for libraries) or poetry/poetry-core (great for apps) too of course. Here's one example from one of my projects where I'm using hatchling. Here's the official guide to get you started: https://packaging.python.org/en/latest/tutorials/packaging-projects/

0xdolan commented 1 year ago

What if we try to clone a private repository from Bitbucket (or GitHub) but using environment variables instead of directly embedding the credentials using a .env file? Is this approach considered a best practice?

for example:

my_private_repo = { git = "https://$USERNAME:$PASSWORD@bitbucket.org/companyrepo/myproject.git", branch = "develop" }

systematicguy commented 1 year ago

What if we try to clone a private repository from Bitbucket (or GitHub) but using environment variables instead of directly embedding the credentials using a .env file? Is this approach considered a best practice?

for example:

my_private_repo = { git = "https://$USERNAME:$PASSWORD@bitbucket.org/companyrepo/myproject.git", branch = "develop" }

We achieve our private git dependency reference using ssh url. IMHO it is more flexible to set up, and more uniform, should you have some other public git dependencies, or should you decide later to relax protection and open up your repo. IMO credentials laying around in files pose a harder-to-justify security risk, because you (your CI) depend on cleaning up after yourself (your build).

We use ssh git urls successfully across windows, wsl, docker, mac environments with acceptable level of inconvenience (mandating passphrase protected ssh keys + ssh agent).

0xdolan commented 1 year ago

I agree, but I think we should have both options on the table. I have tried ssh already, but because of some reasons, using env variables would make more sense in my case and it helps to be more flexible to some extent! BTW, thank you for clarification. I appreciate your time.

GruenSein commented 1 year ago

What if we try to clone a private repository from Bitbucket (or GitHub) but using environment variables instead of directly embedding the credentials using a .env file? Is this approach considered a best practice? for example: my_private_repo = { git = "https://$USERNAME:$PASSWORD@bitbucket.org/companyrepo/myproject.git", branch = "develop" }

We achieve our private git dependency reference using ssh url. IMHO it is more flexible to set up, and more uniform, should you have some other public git dependencies, or should you decide later to relax protection and open up your repo. IMO credentials laying around in files pose a harder-to-justify security risk, because you (your CI) depend on cleaning up after yourself (your build).

We use ssh git urls successfully across windows, wsl, docker, mac environments with acceptable level of inconvenience (mandating passphrase protected ssh keys + ssh agent).

While using ssh and the required key pair works for me as well, it requires to be set up for each user/machine individually, correct? IIRC, my key pair is set up for my personal github acount which in turn needs to be granted access to the dependency repository. For a large number of users, this is quite inconvenient to repeat for each new user. It is much simpler to use a read-only access token, which users and CI pipelines can easily add to their environment variables because it does not require any action per new user by the dependency maintainer. Or am I missing something?

nicolaipre commented 11 months ago

Leaving this here in case it is useful for others.

I got it working running this command:

$ poetry add git+ssh://git@github.com/username/private_repo.git@master

For me it worked after adding git@ in front of github.com. There is also a chance this made an impact:

$ GH_TOKEN="TOKEN" echo "https://${GH_TOKEN}:@github.com" > ${HOME}/.git-credentials && git config --global credential.helper store

I am using Fine-grained Tokens: https://github.com/settings/tokens?type=beta

yordis commented 10 months ago

Hey peeps, I am not a Python dev, I am trying to figure out how to build a docker image using poetry with a private repo, hopefully using Docker buildkit and the ssh mount.

Do you have any implementation reference I could follow?

python-poetry / poetry