eos client in Go - Githubissues

sbinet commented 6 years ago

hi there,

(apologies if that's not the correct way to ask for informations...)

I am wondering whether I couldn't create an EOS client in my favorite language, Go. I also want it to be BSD-3 (so I can't really look at the eos C++ sources).

I suppose, as a first step, I should just focus on interacting with EOS at the REST API level.

any pointers?

(my end goal is threefold: connect EOS with upspin.io and with a Go-based Jupyter kernel, and eventually, connect it with a Go-based xrootd implementation)

ljanyst commented 6 years ago

It would be extremely interesting to me personally to see a Go based implementation of the XRootD client. With the go concurrency model and its support for coroutines it should be much more robust than the C++ implementation and much simpler.

While I was still at CERN, I played with implementing a basic client in pure Python using twisted-style cooperative multithreading. The overall performance was rather pathological because of Python's GIL issues (inability to process callbacks in parallel in the same memory space), but I was amazed by how little code was needed to have basic reads working.

apeters1971 commented 6 years ago

You can safe yourself a lot of work and just use the HTTP server plugin to talk to EOS. No need to write any GO client.

Cheers Andreas.

On Tue, Dec 5, 2017 at 2:59 PM, Sebastien Binet notifications@github.com wrote:

hi there,

(apologies if that's not the correct way to ask for informations...)

I am wondering whether I couldn't create an EOS client in my favorite language, Go https://golang.org. I also want it to be BSD-3 (so I can't really look at the eos C++ sources).

I suppose, as a first step, I should just focus on interacting with EOS at the REST API level.

(my end goal is threefold: connect EOS with upspin.io and with a Go-based Jupyter kernel, and eventually, connect it with a Go-based xrootd implementation)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cern-eos/eos/issues/13, or mute the thread https://github.com/notifications/unsubscribe-auth/ACiGZUhgFV5x322T8GNAg4lVzCFsKlGeks5s9UxfgaJpZM4Q2VSl .

sbinet commented 6 years ago

@apeters1971 yeah, I think that's my first logical step.

say I want to access data on eosuser.cern.ch using e.g. http://eos.readthedocs.io/en/latest/restapi/format.html#user-queries, from outside CERN.

how would I do it?

I tried:

$>  curl  "http://eosuser.cern.ch:8000//proc/user/?mgm.cmd=ls?mgm.format=json"
curl: (7) Failed to connect to eosuser.cern.ch port 8000: Connection timed out

on lxplus, I got:

16:00 binet@lxplus088 ~% curl "http://eosuser.cern.ch:8000//proc/user/?mgm.cmd=ls?mgm.format=json" 
<!DOCTYPE html>
<html lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Language" content="en" />
  <meta content="width=device-width, initial-scale=1, user-scalable=yes"
        name="viewport" />

[...]
<body class="lang-en fluid">
  <!-- THE CERN_BANNER IS REQUIRED BY THE GRAPHIC CHARTER -->
  <div id="cern_banner">
    <a href="http://www.cern.ch/" title="cern.ch">
      <span>CERN &mdash; the European Organization for Nuclear Research</span>
    </a>
  </div>
  <div id="container" class="clear-block">
    <div id="middle" class="clear-block">
      <h1>500 - Unable to give access - user access restricted - unauthorized identity used ; Permission denied</h1>
      <p>
        There was an error loading the page you requested
        <script type="text/javascript">
          document.write(": " + document.location.href);
        </script>
        &mdash; This page may have been deleted or moved.
      </p>
      <ul class="plain airy">
        <li>
          <a href="http://www.cern.ch">CERN homepage</a>
        </li>
        <li>
          <a href="http://itssb.web.cern.ch/">IT service status</a>
        </li>
        <li>
          <a href="https://cern.ch/service-portal/">Contact the Service Desk</a>
        </li>
      </ul>
    </div>
    <!-- / middle -->
    <div id="footer" class="clear-block">
      <p>European Organization for Nuclear Research, CH-1211, Genève
        23, Switzerland</p>
      <!-- THE CERN_LOGO IS REQUIRED BY THE GRAPHIC CHARTER -->
      <a id="cern_logo" class="badge_80" href="http://www.cern.ch"
        title="www.cern.ch" name="cern_logo"><span>cern.ch</span></a>
    </div>
    <!-- / footer -->
  </div>
  <!-- / container -->
</body>
</html>

so, yeah... I haven't managed to provide any authentication (nor do I know how exactly I could do that, assuming I have a valid krb5 token on my lxplus machine...) in one case and timeout in the other case...

sbinet commented 6 years ago

@ljanyst wrt Go+xrootd: I also think it should be a match made in heaven (both for the client and the server parts). time will tell :) (see: https://github.com/go-hep/hep/issues/139)

ljanyst commented 6 years ago

Does it mean that you will work on it? I would actually be interested in spending a little time on it as well - it's an awesome learning experience. I know the XRootD protocol inside-out (designed the C++ client) but only the basic concepts of Go.

sbinet commented 6 years ago

it's indeed something I'll have to provide for my evil plan for HEP world Gomination to succeed. but if somebody beats me to it, that's even better :) (let's follow this up on the go-hep/hep tracker ?)

ljanyst commented 6 years ago

I would be unlikely to beat you. I was primarily interested in looking at the core communication engine and seeing how I could design the asynchronous APIs using the Go concepts. I probably would not even touch the distributed failure handling and recovery, which, honestly, is probably most of the work.

sbinet commented 6 years ago

@apeters1971 apologies for pinging back, but, do you have any insights about using the REST interface from lxplus and with CURL? (ie: https://github.com/cern-eos/eos/issues/13#issuecomment-349331544)

as I said, I can't really look at how the eos client manages to get at either the kerberos token or the X509 certificate b/c I want my client to be BSD-3... sorry about that.

apeters1971 commented 6 years ago

You cannot call direct on that port, this is protected. You have to go through an NGINX proxy with https and a x509 or kerberos token to do this calls. Please write a mail to luca.mascetti@cern.ch to get the name of the NGINX proxy to use. I am not 100% sure if it has to be the CERNBOX portal or if there is a separate NGINX proxy to EOSUSER http access.

On Thu, Dec 7, 2017 at 3:00 PM, Sebastien Binet notifications@github.com wrote:

@apeters1971 https://github.com/apeters1971 apologies for pinging back, but, do you have any insights about using the REST interface from lxplus and with CURL? (ie: #13 (comment) https://github.com/cern-eos/eos/issues/13#issuecomment-349331544)

as I said, I can't really look at how the eos client manages to get at either the kerberos token or the X509 certificate b/c I want my client to be BSD-3... sorry about that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cern-eos/eos/issues/13#issuecomment-349974748, or mute the thread https://github.com/notifications/unsubscribe-auth/ACiGZWGuoi4jvi4BTsln2IxIXD4aWl9yks5s9-90gaJpZM4Q2VSl .

cern-eos / eos

eos client in Go #13