search

opnsense / plugins

OPNsense plugin collection
https://opnsense.org/
BSD 2-Clause "Simplified" License
846 stars 642 forks source link

security/acme-client: some lexicon providers don't work out of the box #1577

Closed nibblerrick closed 4 years ago

nibblerrick commented 5 years ago

Describe the bug I tried to issue a letsencrypt certificate at a Hetzner dns server through lexicon API (dns-01). First attempt threw "please install lexicon" in the logfile. On my system there ist lexicon-3.7 in /usr/local/bin but no lexicon. I made a symlink lexicon pointing to lexicon-3.7 and tried again. This it did more, but no success. So i wanted to try on the console and trying just "lexicon hetzner -h" gives "WARNING: some required dependencies for this provider are not installed. Please install lexicon[hetzner] first before using it.". At this point I thought I'll write this bugreport before doing to many things manually on the system.

Is having lexicon-3.7 but not lexicon as command expected behaviour or a bug? Are providers which need more depencies supposed to be working or not?

Is this the correct repository to file the bugreport?

Environment OPNsense 19.7.6-amd64

fraenki commented 5 years ago

@jpawlowski Since you're the only lexicon expert (lexpert?) I know, could you comment on this please? :)

jpawlowski commented 5 years ago

I think the solution is documented here: https://github.com/opnsense/tools/issues/156

Generally spoken, py-dns-lexicon needs to be installed manually as I believe the dependency from acme.sh package is still missing (didn't check on it). Maybe that dependency is already there. If someone had installed acme.sh earlier before I don't think this dependency will automatically be resolved (unless the acme.sh package will get an update so it will trigger to install dependencies once again). Existing installations might need to manually install py-dns-lexicon just once or re-install the acme.sh package.

fichtner commented 5 years ago

Looks ok to me https://github.com/opnsense/plugins/blob/4bc92be30d3f5e9cbcf44c007344cdafdf1215d3/security/acme-client/Makefile#L5

On 16. Nov 2019, at 10:21, Julian Pawlowski notifications@github.com wrote:

 I think the solution is documented here: opnsense/tools#156

Generally spoken, py-dns-lexicon needs to be installed manually as I believe the dependency from acme.sh package is still missing (didn't check on it).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

jpawlowski commented 5 years ago

So it seems there is additional dependencies for some providers: https://github.com/AnalogJ/lexicon/blob/master/setup.py#L34-L44

For Hetzner it is 'dnspython>=1.15.0', 'beautifulsoup4'. I'll prepare a PR for it.

fichtner commented 5 years ago

It would be nice to lay out all facts before pulling in more packages. There has to be an overview somewhere?

On 16. Nov 2019, at 10:36, Julian Pawlowski notifications@github.com wrote:

 So it seems there is additional dependencies for some providers: https://github.com/AnalogJ/lexicon/blob/master/setup.py#L34-L44

Für Hetzner it is dnspython>=1.15.0', 'beautifulsoup4'. I'll prepare a PR for it

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

jpawlowski commented 5 years ago

We do miss pre-compiled packages for:

In my opinion, it is worth adding those packages from ports:

I would leave out all of the other providers special packages unless someone really needs those.

So, in total, those are the package dependencies to be added to https://github.com/opnsense/plugins/blob/master/security/acme-client/Makefile:

py${PLUGIN_PYTHON}-boto3 py${PLUGIN_PYTHON}-xmltodict py${PLUGIN_PYTHON}-beautifulsoup460 py${PLUGIN_PYTHON}-dnspython

... and to https://github.com/opnsense/tools/blob/master/config/20.1/ports.conf:

devel/py-xmltodict@py${PRODUCT_PYTHON3}
www/py-beautifulsoup460@py${PRODUCT_PYTHON3}

Am I right? Not 100% sure about the dynamic Python versioning stuff ;-)

nibblerrick commented 4 years ago

Thanks for the effort to bring this in the distribution. I'll wait and try when it's ready. Is the problem that "lexicon" gets called but there is only "lexicon-3.7" in /usr/local/bin also related to this?

jpawlowski commented 4 years ago

Is the problem that "lexicon" gets called but there is only "lexicon-3.7" in /usr/local/bin also related to this?

This seems unrelated to me. However, not sure why the symlink would be missing. All my OPNsense instances here had the package installed manually using the OPNsense tools so it is not exactly what would happen to a clean install after the lexicon package was added to the distribution channels. However, build commands are the same so not sure where the missing symlink comes from. Maybe worth opening a separate issue for this.

mdbraber commented 4 years ago

I would like to add a request for the TransIP package.

Also - I'm seeing the same @nibblerrick: the lexicon-3.7 is not symlinked to 'lexicon'.

fichtner commented 4 years ago

The issue should be fixed on 20.1:

# pkg info -l py37-dns-lexicon | grep /lexicon$
    /usr/local/bin/lexicon
mdbraber commented 4 years ago

@fichtner thanks for the fix. I'm trying to figure out how to (manually) install the 'transip' dependency. I see it's already in the OPNsense ports, but not installed. Should I manually compile it using the instructions here or is there an easier way? https://forum.opnsense.org/index.php?topic=15011.msg68842#msg68842

fichtner commented 4 years ago

you may be looking for this https://github.com/opnsense/ports/commit/db62e2d673 which was committed to FreeBSD just recently

So ideally we want to enable the option(s) in tools.git and it takes care of all dependencies automatically. for the moment and testing the manual build is fine...

mdbraber commented 4 years ago

@fichtner thanks for the pointer, I found that commit before, but didn't realise it was just for 20.1. As I don't want to be running cutting edge stuff, I just installed the ports and did a manual build. After correcting for the right Python version it works fine.

fichtner commented 4 years ago

so which option are you looking for? we can integrate it on 20.1.x

mdbraber commented 4 years ago

@fichtner I'm using the TransIP (net/py-transip)options - would be great if those would be integrated!

nibblerrick commented 4 years ago

So I wanted to give it another try and it seems to find lexicon and dns_hetzner.

However it's not successful:

[Sun May 31 14:49:39 CEST 2020] d_api='/usr/local/share/examples/acme.sh/dnsapi/dns_lexicon.sh'
[Sun May 31 14:49:39 CEST 2020] Found domain api file: /usr/local/share/examples/acme.sh/dnsapi/dns_lexicon.sh
[Sun May 31 14:49:39 CEST 2020] Adding txt value: XXXXXX for domain:  _acme-challenge.blah
[Sun May 31 14:49:39 CEST 2020] LEXICON_HETZNER_USERNAME='[hidden](please add '--output-insecure' to see this value)'
[Sun May 31 14:49:39 CEST 2020] LEXICON_HETZNER_TOKEN='[hidden](please add '--output-insecure' to see this value)'
[Sun May 31 14:49:39 CEST 2020] LEXICON_HETZNER_PASSWORD='[hidden](please add '--output-insecure' to see this value)'
[Sun May 31 14:49:39 CEST 2020] LEXICON_HETZNER_DOMAINTOKEN='[hidden](please add '--output-insecure' to see this value)'
[Sun May 31 14:49:39 CEST 2020] LEXICON_HETZNER_API_KEY='[hidden](please add '--output-insecure' to see this value)'
[Sun May 31 14:49:39 CEST 2020] LEXICON_OPTS='[hidden](please add '--output-insecure' to see this value)'
[Sun May 31 14:49:41 CEST 2020] Error add txt for domain:_acme-challenge.blah
[Sun May 31 14:49:41 CEST 2020] _on_issue_err
[Sun May 31 14:49:41 CEST 2020] Please check log file for more details: /var/log/acme.sh.log
[Sun May 31 14:49:41 CEST 2020] url='https://acme-v02.api.letsencrypt.org/acme/chall-v3/...

I copied the /usr/local/share/examples/acme.sh/dnsapi/dns_lexicon.sh to /root/.acme.sh/dnsapi/, set the enviromentvariables PROVIDER and LEXICON_HETZNER_TOKEN and did an acme.sh --issue -d blah --dns dns_lexicon --debug 2 And it went fine. "The txt record is added: Success." instead "Error add txt for domain:_acme-challenge.blah".

I just don't know if this has something to do with the relative new Hetzner API introduction?

fraenki commented 4 years ago

@nibblerrick The next version of os-acme-client will include native support for Hetzner DNS API, no need to use lexicon anymore (see #1870).

fraenki commented 4 years ago

I think the approach of lexicon somewhat contradicts the whole point of acme.sh: while acme.sh tries to eleminate the need for additional packages by implementing most stuff in Posix Shell, lexicon requires additional Python modules for many DNS APIs. As a result, it requires more project resources to support lexicon providers. :-(

I'll leave it this way for now. But in the future we should consider to focus more on implementing all available acme.sh DNS APIs and maybe deprecate support for lexicon after some time.