Synda Certificate Renew Fails

[ericnienhouse]

We've recently had our synda certificate renew fail, always:

$ synda certificate --debug renew Error occurs while renewing certificate ([('SSL routines', 'SSL3_GET_RECORD', 'wrong version number')])

It is unclear what may have changed in our local (or remote server) configuration to affect this. It seems related to protocol support by the myproxy server?

Note we're using: esgf-node.llnl.gov:7512

Note: Adding --debug does not seem to provide additional information about the failure.

[senesis]

The same occurs for me, for a fresh install, at first trial of a 'synda install' command, and on 'synda certificate renew'

[AtefBN]

The error message is pretty cryptic and not a good indication of what went wrong in my experience with synda. But in my previous encounters, it has always been a wrong version of a dependency (openssl or myproxyclient mostly). Can you verify they're in order?

[senesis]

Updating a synda 3.9 DEB install with a synda 3.10 conda install did, in my case, update openssl, and led to a successful 'synda certificate renew'

[hot007]

I had the same problem using the centos71 RPM on a centos7.6 system, which is synda-3.8-1.x86_64. Removed the RPM and tried instead with conda. This was successful and synda seems to be fundamentally working (can install files).

[hot007]

I say "fundamentally" because I've found that in working through the synda intro examples, one that used to work now fails:

> synda search -f cmip5.output1.MPI-M.MPI-ESM-LR.decadal1995.mon.land.Lmon.r2i1p1.v20120529 baresoilFrac
WARNING: 'baresoilFrac' value has been associated with 'variable_id' facet.
WARNING: 'baresoilFrac' value has been associated with 'variable_id' facet.
File not found

Has anyone else found this? As far as I can tell things are otherwise working, I've downloaded a few test files and I'm about to try a selection file and some other options. Just noting something seems to have changed in synda search in the conda version that's causing something odd to happen in that example.

[oloapinivad]

Hi all,

I am starting to use this tool and I found extremely powerful, so first of all thanks to the developers!

I just wanted to mention that I installed synda 3.10 via conda and I was facing the same issue as above. I had to manual downgrade the openssl package from 1.1.1e to 1.1.1d in order to overcome it. I suspect that the current conda installation is not working as it is.

[francocatalano]

Hi all, I am experiencing a similar error. If I do: synda certificate --debug renew I get: Error occurs while renewing certificate (exit) without any additional information about the error. I did not have problems untill january 2021, then I started getting this issue. version of synda: 3.14 version of conda: 4.7.12 Any idea about what's wrong? Thanks a lot in advance for your help!

[francocatalano]

I have tried downgrading openssl and still same problem with certificate. Then I tried removing synda and conda and doing a clean conda/synda reinstall. After that I have: conda version 4.8.3 synda version 3.20 but when I launch synda I got the following error: synda.source.process.subcommand.exceptions.InvalidRequest: Not found Invalid request

Then, I tried to downgrade synda to version 3.14 and got again the certificate error: Error occurs while renewing certificate (exit)

Is there anyone currently able to use synda? If yes, could you please share your configuration? Any information would be very appreciated. Thanks!

[painter1]

In my experience most certificate-related errors have not arisen from Synda, but from problems with the identity server used to update certificates. And most of the time the existing certificate is still valid, so the renewal is not really necessary. I am running a heavily patched older version of Synda, and have submitted a few pull requests for some of the patches. This version of Synda will try to continue even after failing to renew a certificate, and in practice this has always been successful. I will soon submit a pull request for this feature.

Jeff

From: Franco Catalano notifications@github.com Reply-To: Prodiguer/synda reply@reply.github.com Date: Monday, March 1, 2021 at 8:54 AM To: Prodiguer/synda synda@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [Prodiguer/synda] Synda Certificate Renew Fails (#121)

I have tried downgrading openssl and still same problem with certificate. Then I tried removing synda and conda and doing a clean conda/synda reinstall. After that I have: conda version 4.8.3 synda version 3.20 but when I launch synda I got the following error: synda.source.process.subcommand.exceptions.InvalidRequest: Not found Invalid request

Then, I tried to downgrade synda to version 3.14 and got again the certificate error: Error occurs while renewing certificate (exit)

Is there anyone currently able to use synda? If yes, could you please share your configuration? Any information would be very appreciated. Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.us/v2/url?u=https-3A__github.com_Prodiguer_synda_issues_121-23issuecomment-2D788103359&d=DwMCaQ&c=pKoAVQro6qDbLoK0T8588B4mZJhJhC4e6QXJy0XnJec&r=Pyh7ggQUl3TbVyCxvHXd5tS9ZUQzYqbVcelT1fZ78eo&m=rKj8Z8mi2RIH50TRP5Stj3nHjTy0Bd9rAk3r1VqJDMg&s=ihtU8T4wtXU1P5r59Kvv5dyF5qCrAmHwv3pt8fbwqng&e=, or unsubscribehttps://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAVLQMPCUAQOEOATQM2MEI3TBPBD3ANCNFSM4IJPNRIQ&d=DwMCaQ&c=pKoAVQro6qDbLoK0T8588B4mZJhJhC4e6QXJy0XnJec&r=Pyh7ggQUl3TbVyCxvHXd5tS9ZUQzYqbVcelT1fZ78eo&m=rKj8Z8mi2RIH50TRP5Stj3nHjTy0Bd9rAk3r1VqJDMg&s=NleX6RaOLq49VLYVPbdpM_zh42Zgrkl6GPM9JpcQSEg&e=.

[painter1]

Actually I had submitted this patch last year, pull request #145. Nobody paid attention, and since then the master branch was changed so as to make it incompatible. I will submit a new pull request with the same patch, but compatible with the present master branch. I will not revise it again.

[painter1]

A new pull request, $164, does the same thing as #145. To enable it, edit the configuration file: add a line "continue_on_cert_errors=true" in the download section. Then, if Synda tries to renew a certificate while starting transfers, and if the renewal fails, Synda will continue to run. If the certificate hadn't expired yet (which is likely), Synda will transfer data exactly as if the renewal had succeeded.

I run with this feature turned on, 24 hours a day with about 4-8 data nodes simultaneously, 6 files per node. Although it's a tiny and simple change, it has a tremendous effect on reliability.

[francocatalano]

Thanks Jeffrey. How do I get your fix in synda version 3.14? Unfortunately, as I wrote, it seems the new version 3.20 is not working on my server.

[painter1]

The clean way to get it into your version is with a git merge from the continue_on_cert_errors branch to your git-controlled copy of Synda. If you have a separate working copy as I do, you will have to copy the changed files to there. That's four files, the four files containing the string "continue_on_cert_errors".

A dirtier (slightly more dangerous from the long-term maintenance perspective) way is to directly edit these four files in the working copy, in order to match. In my case and probably yours, the working copy of Synda is in /usr/share/python/synda/, in various subdirectories. Three of the four changed files are about getting user settings from sdt.conf and are named contants.py, models.py, and models.py (again!) IIRC. The one which directly affects the logic of what Synda does is sddmdefault.py, typically in /usr/share/python/synda/synda/sdt/.

The super-dirty way to incorporate this change is to bypass the user settings in sdt.conf and instead directly edit /usr/share/python/synda/synda/sdt/sddmdefault.py. You can make it match the sddmdefault.py in the continue_on_cert_errors branch, but replace "preferences.is_download_continue_on_cert_errors" with True. I can't really recommend hacks like that, except for code development purposes.

However you do it, you will have to restart the daemon afterwards.

This is quite a short simple patch, and is intended to keep things going through brief outages of the identity (myproxy) server. If it is down for an extended period of time, you really will need to renew some certificates. Then the only solution is to find another server for renewing certificates.

[francocatalano]

Thanks again Jeffrey, I really appreciate your help! Since, because of this issue, I have not been able to download files with synda in the last month (at least) I fear that I will need to renew my certificate in any case. How do I check which identity (myproxy) server I am using and how can I switch to a different one?

[painter1]

I believe that the server name is extracted from your OpenID which is set in credentials.conf.

[francocatalano]

Thanks Jeffrey. Then I believe the problem is not related to my identity server (esgf-node.llnl.gov) because I am able to download CMIP files using esgf-generated wget scripts without any certificate issues. Therefore, I think the problem must be related to synda. Indeed, I am not even able to start synda daemon. If I do: synda daemon start I see the following error in transfer log: 2021-03-04 08:01:18,320 INFO SDMYPROX-002 Renew certificate.. 2021-03-04 08:01:18,489 ERROR SYDLOGON-012 Error occured while retrieving certificate from myproxy server (exit) As I wrote, I have tried with fresh reinstall of different synda versions (3.12, 3.13, 3.14) and got the same error while with latest v3.20 I got errors even when initialising synda environment. It's a pity because synda is a very useful software and getting all the data we need in our lab for CMIP6 analysis without synda would be a nightmare. Thanks again for your effort.

[painter1]

You are right. I also use esgf-node.llnl.gov. It only has an occasional brief failure, possibly due to overloading. My little patch deals with this successfully but couldn't possibly help your situation. Can you run myproxy-logon manually, using the OpenID and password in your credentials.conf?

[francocatalano]

Unfortunately, myproxy-logon is not installed on our server. When I use esgf-generated wget scripts, certificate is obtained with java (getcert.jar) and stored in my $HOME/.esg/credentials.pem. The OpenID and password I pass to the wget script are exactly the same as those specified in synda credentials.conf.

This is the output: Retrieving Federation Certificates...--2021-03-05 13:44:25-- https://github.com/ESGF/esgf-dist/raw/master/installer/certs/esg-truststore.ts Risoluzione di github.com (github.com)... 140.82.121.4 Connessione a github.com (github.com)|140.82.121.4|:443... connesso. Richiesta HTTP inviata, in attesa di risposta... 302 Found Posizione: https://raw.githubusercontent.com/ESGF/esgf-dist/master/installer/certs/esg-truststore.ts [segue] --2021-03-05 13:44:26-- https://raw.githubusercontent.com/ESGF/esgf-dist/master/installer/certs/esg-truststore.ts Risoluzione di raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.110.133, ... Connessione a raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connesso. Richiesta HTTP inviata, in attesa di risposta... 200 OK

Then it asks for my openid and password Retrieving Credentials...mar 05, 2021 1:45:32 PM esg.security.myproxy.CredentialConnection getCredential INFO: done! done!

and the download starts. I don't know why synda is failing to get certificate. Thanks again for your help.

[plesager]

@francocatalano were you able to create a certificate finally? If you did, which version of synda are you using?

[francocatalano]

@plesager Hi Philippe. After many attempts, I managed to get it working with synda v3.32. So far, it seems to work... good luck!

[plesager]

Thanks Franco. I was trying my openID created by CEDA (it adds an extra username between enduser and ESGF) which gave the same certificate problem as you had. I've created another openID on esgf-node.llnl.gov and that fixes it (synda 3.35 installed through conda).