Quobyte package version "1.0.1b0" breaks generation of offline Universe

[fersantxez]

When generating an offline universe as explained in: https://docs.mesosphere.com/1.8/administration/installing/custom/deploying-a-local-dcos-universe/#scrollNav-2 An error was detected (full log following below): ValueError: unknown url type: 'dcos-quobyte>=1.0.1b0' Removing the quobyte package and applying this list solved it: arangodb3,artifactory,cassandra,chronos,confluent-kafka,dse,gitlab,jenkins,marathon,spark,arangodb,hdfs,kafka,marathon-lb,nginx,wordpress,zeppelin ...so it looks to me like the quobyte package may be broken. eded68eb55ea: Pushed e5cb163894f9: Pushed e5cb163894f9: Pushed dd3463e7917c: Pushed dd3463e7917c: Pushed 9dab486f5005: Pushed 9dab486f5005: Pushed 7e6dc736eb81: Pushed 7e6dc736eb81: Pushed e01f5b4e291f: Pushed e01f5b4e291f: Pushed c2e82d3fcfc2: Pushed c2e82d3fcfc2: Pushed fa9685041e05: Pushed fa9685041e05: Pushed 13400a2548ac: Pushed 13400a2548ac: Pushed 135a7d6b5fb0: Pushed 135a7d6b5fb0: Pushed 6eb35183d3b8: Pushed 6eb35183d3b8: Pushed ae30bba0ede8: Pushed ae30bba0ede8: Pushed dcos-latest: digest: sha256:10566d82a1686c3513dc96244dabd6ac7eeedac760fb18105ea53ff147119acf size: 5509 registry Traceback (most recent call last): File "/root/universe/docker/local-universe/../../scripts/local-universe.py", line 314, in Adding https://downloads.mesosphere.com/zeppelin/assets/icon-service-zeppelin-large.png at /tmp/tmpvai65s2c/http/zeppelin/images/icon-service-zeppelin-large.png. Adding https://downloads.mesosphere.com/zeppelin/assets/icon-service-zeppelin-small.png at /tmp/tmpvai65s2c/http/zeppelin/images/icon-service-zeppelin-small.png. Adding https://downloads.mesosphere.io/spark/assets/spark-1.6.0.tgz at /tmp/tmpvai65s2c/http/zeppelin/uris/spark-1.6.0.tgz. Pull docker images: mesosphere/spark:1.6.0 Pushing docker image: mesosphere/spark:1.6.0 Pull docker images: galacticfog/gestalt-dcos:latest Pushing docker image: galacticfog/gestalt-dcos:latest Pull docker images: mesosphere/zeppelin:0.5.6-3 Pushing docker image: mesosphere/zeppelin:0.5.6-3 Pull docker images: galacticfog/lambda-javascript-executor:dcos-latest Pushing docker image: galacticfog/lambda-javascript-executor:dcos-latest Pull docker images: galacticfog/gestalt-api-gateway:dcos-latest Pushing docker image: galacticfog/gestalt-api-gateway:dcos-latest Pull docker images: galacticfog/kong:dcos-latest Pushing docker image: galacticfog/kong:dcos-latest Pull docker images: galacticfog/lambda-dotnet-executor:dcos-latest Pushing docker image: galacticfog/lambda-dotnet-executor:dcos-latest Pull docker images: galacticfog/gestalt-data:dcos-latest Pushing docker image: galacticfog/gestalt-data:dcos-latest Pull docker images: galacticfog/gestalt-lambda:dcos-latest Pushing docker image: galacticfog/gestalt-lambda:dcos-latest Completed: gestalt-framework Completed: gitlab Completed: hdfs Completed: hue Completed: jenkins Completed: kafka-manager Completed: kafka Completed: kibana Completed: linkerd-viz Completed: linkerd Completed: logstash Completed: marathon-lb Completed: marathon-slack Completed: marathon Completed: mariadb Completed: memsql Completed: mr-redis Completed: mysql-admin Completed: mysql Completed: namerd Completed: netsil-collectors Completed: netsil Completed: nginx Completed: omsdocker Completed: openldap-admin Completed: openldap Completed: openvpn-admin Completed: openvpn Completed: postgresql-admin Completed: postgresql Stopping docker registry. sys.exit(main()) File "/root/universe/docker/local-universe/../../scripts/local-universe.py", line 104, in main args.selected)): File "/usr/lib64/python3.4/concurrent/futures/_base.py", line 549, in result_iterator yield future.result() File "/usr/lib64/python3.4/concurrent/futures/_base.py", line 395, in result return self.get_result() File "/usr/lib64/python3.4/concurrent/futures/_base.py", line 354, in get_result raise self._exception File "/usr/lib64/python3.4/concurrent/futures/thread.py", line 54, in run result = self.fn(_self.args, *_self.kwargs) File "/root/universe/docker/local-universe/../../scripts/local-universe.py", line 87, in handle_package add_http_resource(http_artifacts, url, archive_path) File "/root/universe/docker/local-universe/../../scripts/local-universe.py", line 245, in add_http_resource urllib.request.urlretrieve(url, str(archive_path)) File "/usr/lib64/python3.4/urllib/request.py", line 186, in urlretrieve with contextlib.closing(urlopen(url, data)) as fp: File "/usr/lib64/python3.4/urllib/request.py", line 161, in urlopen return opener.open(url, data, timeout) File "/usr/lib64/python3.4/urllib/request.py", line 448, in open req = Request(fullurl, data) File "/usr/lib64/python3.4/urllib/request.py", line 266, in init self.full_url = url File "/usr/lib64/python3.4/urllib/request.py", line 292, in full_url self._parse() File "/usr/lib64/python3.4/urllib/request.py", line 321, in _parse raise ValueError("unknown url type: %r" % self.full_url) ValueError: unknown url type: 'dcos-quobyte>=1.0.1b0' make: *\ [local-universe] Error 1

[fersantxez]

@casusbelli ^^

[casusbelli]

Thanks for the hint, will check this out

[casusbelli]

Ok, i can reproduce this easily. Seemingly version-3.x branch docker build cannot interpret the command.json pip field in pips requirements.txt format but only as an url. Since that config file is deprecated in version-3.x anyway I'll move it to resource config as described in the docs.

[casusbelli]

Fix at #756 but different from my previous comment this still uses command.json as the version 3 based format requires the binary client which currently is not available.

[joerg84]

@casusbelli Hi, is this done and ready to land? @ryadav88 just FYI

[casusbelli]

@joerg84 Hi! Change can be merged, there seemingly was a CI hiccup but that seems to be solved now and the change was tested ok. So far it has not been merged, however.

[tobilg]

As the Quobyte package has apparently landed, can this be closed? @joerg84 @ryadav88

[casusbelli]

@tobilg hmm, no. There are open reviewers comments i still have to address.