Cannot use subprojects: Meson wrap seems to be down

[germandiagogomez]

Describe the bug

Wrap db not working. Cannot download projects and cannot use locally.

To Reproduce

This simple line won't work:

1. Type meson wrap search sdl

Expected behavior

Meson wrap should download packages for subprojects

Error message in Ubuntu Linux 19.10 (Windows error is the same, but with a bit different output)

SSL connection failed. Falling back to unencrypted connections.
Traceback (most recent call last):
  File "/usr/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/lib/python3.7/http/client.py", line 938, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/usr/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/mesonbuild/mesonmain.py", line 129, in run
    return options.run_func(options)
  File "/usr/local/lib/python3.7/dist-packages/mesonbuild/wrap/wraptool.py", line 216, in run
    options.wrap_func(options)
  File "/usr/local/lib/python3.7/dist-packages/mesonbuild/wrap/wraptool.py", line 78, in search
    jd = get_result(API_ROOT + 'query/byname/' + name)
  File "/usr/local/lib/python3.7/dist-packages/mesonbuild/wrap/wraptool.py", line 57, in get_result
    u = open_wrapdburl(urlstring)
  File "/usr/local/lib/python3.7/dist-packages/mesonbuild/wrap/wrap.py", line 72, in open_wrapdburl
    return urllib.request.urlopen(urlstring, timeout=req_timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 1345, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 113] No route to host>

system parameters

• Is this a cross build or just a plain native build (for the same computer)? No
• what operating system (e.g. MacOS Catalina, Windows 10, CentOS 8.0, Ubuntu 18.04, etc.). Any. I tried Ubuntu Linux 19.10, Windows 10 and Mac Catalina
• what Python version are you using e.g. 3.8.0 3.7.5
• what meson --version 0.52.1
• what ninja --version if it's a Ninja build 1.9.0

[hd1ex]

That is true, https://wrapdb.mesonbuild.com/ is currently down.

[RafaGago]

Wow, just to get git workflows working for the first time in one of my solo repos I was repeatedly force-pushing on a development branch. So there is one Github-generated reference here for each force-push I did because the commit message contained a link to this issue. I was unaware of this.

I tried to delete the references here with no success. There is no button to delete them. Sorry.

[lgrahl]

Ping (I know, everyone hates pings - sorry!). Can we get this fixed? It is also breaking a couple of AUR repos.

[tp-m]

it's being worked on

[xclaesse]

For everyone who has a failing CI because of this: I would recommend against depending on external resources for your CI in general, and on wrapdb in particular since it has been unreliable in the past too.

• You can commit your .wrap files into your git repo
• You can cache subprojects checkout/tarballs into your CI docker/runner. One way of doing that is to run meson subprojects download in your Dockerfile, then copy subprojects/ folder somewhere in your docker image. Then when running the CI job, you can first copy the cached subprojects/ folder.

In the meantime, tarballs can be found in https://github.com/mesonbuild/THE_PROJECT/releases

[eli-schwartz]

For everyone who is more concerned about their project being unbuildable by users, regardless of tricks used in order to make CI pass for a while:

You can and should update the patch_url in your wrapdb-provided .wraps to point to the original file location, which will be in https://github.com/mesonbuild/${name-of-wrapped-project}/releases

The wrapdb maintainers erroneously consider that it is "more reliable" to host independently of github, in case meson ever migrates somewhere else, but this is not the first time the wrapdb has gone down, and I think I know which one I trust more.

Since mesonbuild will not delete all their repos (thus removing historical, archived data) -- right??? -- this is perfectly safe and can't stop working. Although I do understand the desire to ensure meson wrap search|install|update (a developer tool) is independent of github, this doesn't apply to the actual downloaded .wrap files (used by end users via meson setup builddir/ && ninja -C builddir/).

See https://github.com/falconindy/auracle/pull/69 for an example of how to do this for the "fmt" wrap.

[xclaesse]

@eli-schwartz note that is has nothing to do with "more reliable" at all. The point is to have control over the domain name, to not be locked on github.com forever.

[xclaesse]

Since mesonbuild will not delete all their repos (thus removing historical, archived data) -- right??? -- this is perfectly safe and can't stop working.

github is owned by Microsoft, they can change terms, add adds, whatever... Many reasons why we could some day have to delete github account. Not saying this is likely, but could happen, it wouldn't be the first time in internet history that big companies screw up services and plainly shut it down.

[eli-schwartz]

Even if GitHub.com Inc. shut down and decommissioned their website, breaking all downloads, wraps hosted on wrapdb.mesonbuild.com but pointing to patch_url's on github could still be updated to use new hosting, and meson wrap update could allow people to install a new version of the wrap with the same contents, but a different download url.

Making the service unreliable today, in order to protect against it becoming unreliable in the future, is simple illogical.

[eli-schwartz]

A better design for downloading artifacts would be to host a static structure on github, e.g. rely on ${domain}/mesonbuild/${wrap}/releases/download/${version}/foo.zip perhaps? and then make download.mesonbuild.com be a DNS redirect to github.com, allowing you to reimplement it elsewhere as an ftp server in an emergency.

Or rebuild a gh-pages site periodically. You could use e.g. mesonbuild.com/${project}/ pointing mesonbuild.github.io/${project}/ which is each wrap repository's gh-pages branch, I think.

[xclaesse]

(hmm, I wrote a comment about using github pages, but it seems it got lost)

Currently wrapdb needs to run a python script on the host, but that shouldn't be needed. As far as I know the only needed feature is list all available wraps with their version, that can be served as a static text file that gets regenerated when pushing new wraps. So using github pages should be possible to have URLs under mesonbuild.com domain, I think.

[tp-m]

I have a server ready to go, if anyone wants to help set it up let me know.

My suggestion would be to first restore the existing service as it was and then focus on where to go in future.

[xclaesse]

@tp-m totally agree... But I don't think we even have a DB dump anywhere... :/

[germandiagogomez]

I would not like to sound unnice, since this is open-source and all voluntary work, but using wrap and having a failure like this for almost one week seems to me like a high bar to adoption.

This impacted my workflow heavily (yes, I was maybe erroneously relying on wrapdb when using Jenkins pipeline).

There are some mitigations for the problem in the comments above, but I think it would be a good idea to have a proposal to make guarantees and/or fallbacks?

It comes to mind for me that if the concern about github hosting of packages is a problem, why not do the following (in the wrap logic in python):

1. use the wrap service
2. use github as a fallback transparently (maybe with some kind of switch in meson setup)
3. being able to use an option to use a subprojects cached data --> this gets copied into your project when the project is setup.

This way we could guarantee 100% availability for the time being with no additional cost, I understand. In all fairness, there are mitigations mentioned in the guide, but the last thing that someone (like me) with a pipeline wants is that it gets difficult to modify things depending on when a service fails or not, and since this can happen, I would propose to have some behavior similar to the one I drafted above implemented for minimal switching costs.

so for example:

meson setup --wrap-fallback-to=https://github.com/wrap-projects/,/my/subprojects-pool/

How would this behave?

1. redirect wraps and point them to use fallback if mesonwrap db does not work. Security-wise, not sure if this is a good idea without a warning of some kind.
2. in case it works, we are done.
3. in case it does not, it continues with the subprojects directory, that has the cached files as found by the normal wrap. The files get uncompressed and everything works as usual.

Feedback and refinements are welcome.

[legeana]

The project was already switched to a cached proxy mode to github. The plan is to have multiple independent mirrors.

Also I guess a with regard to a long unavailability it is a good idea to setup some better monitoring, I personally wasn't aware of it until Friday the 24th. I will try to fix this in the coming week.