search

datopian / datahub-qa

:package: Bugs, issues and suggestions for datahub.io
https://datahub.io/
32 stars 6 forks source link

Revision processing fails for personal and organization accounts #283

Closed sglavoie closed 8 months ago

sglavoie commented 3 years ago

When I use data-cli, I want to be able to data push with that tool successfully so I can publish my dataset to DataHub.

DataHub shows the message Processing this revision has failed. when pushing a dataset. See Analysis section below for more on this problem.

Acceptance

Tasks

Analysis

Organization accounts are set up locally like so before attempting to push:

export DATAHUB_JSON=~/.config/datahub/config.json

Example content for ~/.config/datahub/config.json:

{
  "token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxaWQiOiJ0ZXN0IiwicGVybWlzc2lvbnMiOiIqIiwic2VydmljZSI6Ixxxxxxxxxx.xxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxx",
  "profile": {
    "id": "test",
    "username": "test"
  }
}

Personal accounts are set up similarly with export DATAHUB_JSON, the content of the configuration file being populated when running data login and logging in with Google.

data push succeeds and a new revision appears on DataHub (but fails to validate). Here's the output of data push --debug for a dataset that was successfully pushed before (e.g. see the dataset here https://datahub.io/core/unicode-emojis):

> [debug] Getting rawstore upload creds
> [debug] Getting authz token for rawstore service
> [debug] Calling rawstore authorize with
> [debug] {
  "metadata": {
    "owner": "test",
    "findability": "published"
  },
  "filedata": {
    "archive/emojis.csv": {
      "length": 471617,
      "md5": "PYg4sBctHL9wtlQhBA5TCQ==",
      "name": "emojis_original"
    },
    "data/validation_report.json": {
      "length": 626,
      "md5": "BSKlIQLI08KZpw3gxusDAw==",
      "name": "validation_report"
    },
    "data/emojis_csv.csv": {
      "length": 471617,
      "md5": "PYg4sBctHL9wtlQhBA5TCQ==",
      "name": "emojis_csv"
    },
    "data/emojis_json.json": {
      "length": 1026404,
      "md5": "VJaQT6vB3k20jahoI5Id4Q==",
      "name": "emojis_json"
    },
    "README.md": {
      "length": 1576,
      "md5": "34J2Qrj4SPPlnWdJzf+E5Q==",
      "name": "readme"
    },
    "datapackage.json": {
      "length": 6349,
      "md5": "0gJzaT/cetPT8GHz64BF/w==",
      "name": "datapackage.json"
    }
  }
}
> [debug] Uploading to rawstore with creds ...
> [debug] {
  "archive/emojis.csv": {
    "upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
    "upload_query": {
      "acl": "public-read",
      "Content-MD5": "PYg4sBctHL9wtlQhBA5TCQ==",
      "Content-Type": "text/plain",
      "key": "3d8838b0172d1cbf70b65421040e5309.csv",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
      "x-amz-date": "20210202T193529Z",
      "policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIlBZZzRzQmN0SEw5d3RsUWhCQTVUQ1E9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjNkODgzOGIwMTcyZDFjYmY3MGI2NTQyMTA0MGU1MzA5LmNzdiJ9LCB7IngtYW16LWFsZ29yaXRobSI6ICJBV1M0LUhNQUMtU0hBMjU2In0sIHsieC1hbXotY3JlZGVudGlhbCI6ICJBS0lBVUNYSllIMlVDSVlDVkVGTS8yMDIxMDIwMi91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0sIHsieC1hbXotZGF0ZSI6ICIyMDIxMDIwMlQxOTM1MjlaIn1dfQ==",
      "x-amz-signature": "77689e63b628d81aac4fc5e65061adb91a8c21ac90c776d6933a0ea78e221993"
    },
    "exists": true
  },
  "data/validation_report.json": {
    "upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
    "upload_query": {
      "acl": "public-read",
      "Content-MD5": "BSKlIQLI08KZpw3gxusDAw==",
      "Content-Type": "text/plain",
      "key": "0522a52102c8d3c299a70de0c6eb0303.json",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
      "x-amz-date": "20210202T193529Z",
      "policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIkJTS2xJUUxJMDhLWnB3M2d4dXNEQXc9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjA1MjJhNTIxMDJjOGQzYzI5OWE3MGRlMGM2ZWIwMzAzLmpzb24ifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQVVDWEpZSDJVQ0lZQ1ZFRk0vMjAyMTAyMDIvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTAyMDJUMTkzNTI5WiJ9XX0=",
      "x-amz-signature": "7a459a6d81ddb9536852715d2d749e9c78bd3e93406907b536e7bc8ba90d332b"
    },
    "exists": true
  },
  "data/emojis_csv.csv": {
    "upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
    "upload_query": {
      "acl": "public-read",
      "Content-MD5": "PYg4sBctHL9wtlQhBA5TCQ==",
      "Content-Type": "text/plain",
      "key": "3d8838b0172d1cbf70b65421040e5309.csv",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
      "x-amz-date": "20210202T193529Z",
      "policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIlBZZzRzQmN0SEw5d3RsUWhCQTVUQ1E9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjNkODgzOGIwMTcyZDFjYmY3MGI2NTQyMTA0MGU1MzA5LmNzdiJ9LCB7IngtYW16LWFsZ29yaXRobSI6ICJBV1M0LUhNQUMtU0hBMjU2In0sIHsieC1hbXotY3JlZGVudGlhbCI6ICJBS0lBVUNYSllIMlVDSVlDVkVGTS8yMDIxMDIwMi91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0sIHsieC1hbXotZGF0ZSI6ICIyMDIxMDIwMlQxOTM1MjlaIn1dfQ==",
      "x-amz-signature": "77689e63b628d81aac4fc5e65061adb91a8c21ac90c776d6933a0ea78e221993"
    },
    "exists": true
  },
  "data/emojis_json.json": {
    "upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
    "upload_query": {
      "acl": "public-read",
      "Content-MD5": "VJaQT6vB3k20jahoI5Id4Q==",
      "Content-Type": "text/plain",
      "key": "5496904fabc1de4db48da86823921de1.json",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
      "x-amz-date": "20210202T193529Z",
      "policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIlZKYVFUNnZCM2syMGphaG9JNUlkNFE9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjU0OTY5MDRmYWJjMWRlNGRiNDhkYTg2ODIzOTIxZGUxLmpzb24ifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQVVDWEpZSDJVQ0lZQ1ZFRk0vMjAyMTAyMDIvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTAyMDJUMTkzNTI5WiJ9XX0=",
      "x-amz-signature": "7580cd6714890c0cc6c1f69ecfd0964d9da7a4fe6733e31abb191bf241b83161"
    },
    "exists": true
  },
  "README.md": {
    "upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
    "upload_query": {
      "acl": "public-read",
      "Content-MD5": "34J2Qrj4SPPlnWdJzf+E5Q==",
      "Content-Type": "text/plain",
      "key": "df827642b8f848f3e59d6749cdff84e5.md",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
      "x-amz-date": "20210202T193529Z",
      "policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIjM0SjJRcmo0U1BQbG5XZEp6ZitFNVE9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogImRmODI3NjQyYjhmODQ4ZjNlNTlkNjc0OWNkZmY4NGU1Lm1kIn0sIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwgeyJ4LWFtei1jcmVkZW50aWFsIjogIkFLSUFVQ1hKWUgyVUNJWUNWRUZNLzIwMjEwMjAyL3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QifSwgeyJ4LWFtei1kYXRlIjogIjIwMjEwMjAyVDE5MzUyOVoifV19",
      "x-amz-signature": "d30a5efa4af4c0878a382b189a9c640529544c5731051406e7a54bd40136cff7"
    },
    "exists": true
  },
  "datapackage.json": {
    "upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
    "upload_query": {
      "acl": "public-read",
      "Content-MD5": "0gJzaT/cetPT8GHz64BF/w==",
      "Content-Type": "text/plain",
      "key": "d20273693fdc7ad3d3f061f3eb8045ff.json",
      "x-amz-algorithm": "AWS4-HMAC-SHA256",
      "x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
      "x-amz-date": "20210202T193529Z",
      "policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIjBnSnphVC9jZXRQVDhHSHo2NEJGL3c9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogImQyMDI3MzY5M2ZkYzdhZDNkM2YwNjFmM2ViODA0NWZmLmpzb24ifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQVVDWEpZSDJVQ0lZQ1ZFRk0vMjAyMTAyMDIvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTAyMDJUMTkzNTI5WiJ9XX0=",
      "x-amz-signature": "d156ea79d155836ee5883a213ca4351fc4fc062248a5e60eb848c9f39970aed2"
    },
    "exists": true
  }
}
> [debug] Uploads to rawstore: Complete
> [debug] Uploading to source spec store
> [debug] Getting authz token for rawstore service
> [debug] Calling source upload with spec
> [debug] {
  "meta": {
    "version": 1,
    "ownerid": "test",
    "owner": "test",
    "dataset": "unicode-emojis",
    "findability": "published"
  },
  "inputs": [
    {
      "kind": "datapackage",
      "url": "https://s3.amazonaws.com/rawstore.datahub.io/d20273693fdc7ad3d3f061f3eb8045ff.json",
      "parameters": {
        "resource-mapping": {
          "archive/emojis.csv": "https://s3.amazonaws.com/rawstore.datahub.io/3d8838b0172d1cbf70b65421040e5309.csv",
          "data/emojis_csv.csv": "https://s3.amazonaws.com/rawstore.datahub.io/3d8838b0172d1cbf70b65421040e5309.csv",
          "data/emojis_json.json": "https://s3.amazonaws.com/rawstore.datahub.io/5496904fabc1de4db48da86823921de1.json",
          "data/validation_report.json": "https://s3.amazonaws.com/rawstore.datahub.io/0522a52102c8d3c299a70de0c6eb0303.json",
          "README.md": "https://s3.amazonaws.com/rawstore.datahub.io/df827642b8f848f3e59d6749cdff84e5.md"
        },
        "descriptor": {
          "bytes": 2443457,
          "count_of_rows": 4590,
          "datahub": {
            "created": "2020-09-24T12:39:55.385717",
            "findability": "published",
            "flowid": "722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/2",
            "modified": "2021-02-02T16:18:05.505508",
            "owner": "sebastien.lavoie",
            "ownerid": "722d7c8098a51a04a1d3a5d41fc9f11c"
          },
          "descriptor": "List of emojis available from the Unicode Consortium. More information can be found in the Unicode® Technical Standard #51.",
          "hash": "60e1206a07ee5bb5e89dcc6b627f9fa5",
          "id": "722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis",
          "keywords": [
            "unicode",
            "emojis",
            "emoji",
            "51",
            "standard",
            "uts"
          ],
          "licenses": [
            {
              "name": "ODC-PDDL-1.0",
              "path": "http://opendatacommons.org/licenses/pddl/",
              "title": "Open Data Commons Public Domain Dedication and License v1.0"
            }
          ],
          "name": "unicode-emojis",
          "profile": "data-package",
          "resources": [
            {
              "path": "archive/emojis.csv",
              "pathType": "local",
              "name": "emojis_original",
              "format": "csv",
              "mediatype": "text/csv",
              "bytes": 471617,
              "datahub": {
                "type": "original"
              },
              "dialect": {
                "caseSensitiveHeader": false,
                "delimiter": ",",
                "doubleQuote": true,
                "header": true,
                "lineTerminator": "\r\n",
                "quoteChar": "\"",
                "skipInitialSpace": false
              },
              "dpp:streamedFrom": "https://s3.amazonaws.com/rawstore.datahub.io/3d8838b0172d1cbf70b65421040e5309.csv",
              "encoding": "UTF-8",
              "hash": "3d8838b0172d1cbf70b65421040e5309",
              "profile": "tabular-data-resource",
              "schema": {
                "fields": [
                  {
                    "format": "default",
                    "name": "Group",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Subgroup",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "CodePoint",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Status",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Representation",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Name",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Section",
                    "type": "string"
                  }
                ],
                "missingValues": [
                  ""
                ]
              }
            },
            {
              "path": "data/validation_report.json",
              "pathType": "local",
              "name": "validation_report",
              "format": "json",
              "mediatype": "application/json",
              "bytes": 626,
              "datahub": {
                "type": "derived/report"
              },
              "description": "Validation report for tabular data",
              "dpp:streamedFrom": "/tmp/tmpjfdpj8_x/722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/validation_report/data/0522a52102c8d3c299a70de0c6eb0303/validation_report.json",
              "hash": "0522a52102c8d3c299a70de0c6eb0303",
              "profile": "data-resource",
              "encoding": "ISO-8859-1"
            },
            {
              "path": "data/emojis_csv.csv",
              "pathType": "local",
              "name": "emojis_csv",
              "format": "csv",
              "mediatype": "text/csv",
              "bytes": 471617,
              "datahub": {
                "derivedFrom": [
                  "emojis"
                ],
                "type": "derived/csv"
              },
              "delimiter": ",",
              "dialect": {
                "caseSensitiveHeader": false,
                "delimiter": ",",
                "doubleQuote": true,
                "header": true,
                "lineTerminator": "\r\n",
                "quoteChar": "\"",
                "skipInitialSpace": false
              },
              "dpp:streamedFrom": "/tmp/tmpjfdpj8_x/722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/emojis_csv/data/2ed376f1c761e6d2936435e1addbf0ca/emojis_csv.csv",
              "encoding": "UTF-8",
              "hash": "2ed376f1c761e6d2936435e1addbf0ca",
              "profile": "tabular-data-resource",
              "quotechar": "\"",
              "rowcount": 4590,
              "schema": {
                "fields": [
                  {
                    "format": "default",
                    "name": "Group",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Subgroup",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "CodePoint",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Status",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Representation",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Name",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Section",
                    "type": "string"
                  }
                ],
                "missingValues": [
                  ""
                ]
              },
              "signed": true
            },
            {
              "path": "data/emojis_json.json",
              "pathType": "local",
              "name": "emojis_json",
              "format": "json",
              "mediatype": "text/csv",
              "bytes": 1026404,
              "datahub": {
                "derivedFrom": [
                  "emojis"
                ],
                "type": "derived/json"
              },
              "delimiter": ",",
              "dialect": {
                "caseSensitiveHeader": false,
                "delimiter": ",",
                "doubleQuote": true,
                "header": true,
                "lineTerminator": "\r\n",
                "quoteChar": "\"",
                "skipInitialSpace": false
              },
              "dpp:streamedFrom": "/tmp/tmpjfdpj8_x/722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/emojis_json/data/5496904fabc1de4db48da86823921de1/emojis_json.json",
              "encoding": "ISO-8859-1",
              "hash": "5496904fabc1de4db48da86823921de1",
              "profile": "tabular-data-resource",
              "quotechar": "\"",
              "rowcount": 4590,
              "schema": {
                "fields": [
                  {
                    "format": "default",
                    "name": "Group",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Subgroup",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "CodePoint",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Status",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Representation",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Name",
                    "type": "string"
                  },
                  {
                    "format": "default",
                    "name": "Section",
                    "type": "string"
                  }
                ],
                "missingValues": [
                  ""
                ]
              },
              "signed": true
            },
            {
              "path": "README.md",
              "pathType": "local",
              "name": "readme",
              "format": "md",
              "mediatype": "text/markdown",
              "bytes": 1576,
              "dpp:streamedFrom": "/tmp/tmpotcydkuk/README.md",
              "encoding": "UTF-8"
            }
          ],
          "sources": [
            {
              "name": "unicode-emoji",
              "path": "https://unicode.org/Public/emoji/latest/emoji-test.txt",
              "title": "UTS #51 Unicode Emoji"
            }
          ],
          "title": "UTS #51 Unicode Emoji",
          "readme": "List of emojis available from the Unicode Consortium. More information can be found in the [Unicode® Technical Standard #51](https://unicode.org/reports/tr51/).\n\n# Data\n\nThe data comes from [this directory at Unicode.org](https://unicode.org/Public/emoji/latest/) as a set of plain text files.\n\n\n# Sources\n\n- The Unicode Consortium: https://unicode.org/Public/emoji/latest/\n\n# Preparation\n\nPython 3.6+ is required. First, install dependencies with the following command:\n\n    pip install -r scripts/requirements.txt\n\n\nThen, run the script from this directory in the following way:\n\n    python scripts/process.py\n\n\n# License\n\nThis Data Package is licensed by its maintainers under the Public Domain Dedication and License (PDDL).\n\n> Copyright © 1991-2017 Unicode, Inc. All rights reserved.\n> Distributed under the Terms of Use in http://www.unicode.org/copyright.html.\n> \n> Permission is hereby granted, free of charge, to any person obtaining a copy of the Unicode data files and any associated documentation (the \"Data Files\") or Unicode software and any associated documentation (the \"Software\") to deal in the Data Files or Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, and/or sell copies of the Data Files or Software, and to permit persons to whom the Data Files or Software are furnished to do so, provided that either (a) this copyright and permission notice appear with all copies of the Data Files or Software, or (b) this copyright and permission notice appear in associated Documentation.\n"
        }
      }
    }
  ],
  "processing": [
    {
      "input": "emojis_original",
      "output": "emojis_original",
      "tabulator": {
        "delimiter": ",",
        "quotechar": "\""
      }
    },
    {
      "input": "emojis_csv",
      "output": "emojis_csv",
      "tabulator": {
        "delimiter": ",",
        "quotechar": "\""
      }
    },
    {
      "input": "emojis_json",
      "output": "emojis_json",
      "tabulator": {
        "delimiter": ",",
        "quotechar": "\""
      }
    }
  ]
}
> [debug] Getting authz token for source service
> [debug] {
  "dataset_id": "test/unicode-emojis",
  "errors": [],
  "flow_id": "test/unicode-emojis/4",
  "success": true
}

🙌  your data is published!
🔗  https://datahub.io/test/unicode-emojis/v/4 (copied to clipboard)

Opening the resources, e.g. https://s3.amazonaws.com/rawstore.datahub.io/d20273693fdc7ad3d3f061f3eb8045ff.json shows they are successfully uploaded. However, DataHub shows this:

https://datahub.io/sebastien.lavoie/unicode-emojis/v/7 (the code from this error can be found here https://github.com/datopian/assembler/blob/master/datapackage_pipelines_assembler/processors/load_private_resource.py#L19)

Processing this revision has failed. See below for details:
Copying source data

assembler.load_private_resource

Traceback (most recent call last):

File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/load_private_resource.py", line 35, in <module>

PrivateResourceLoader()()

File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/load_private_resource.py", line 19, in __init__

if PKGSTORE_BUCKET in url:

TypeError: argument of type 'NoneType' is not iterable

This is not a one-time occurrence as can be seen here:

For some unknown reason, https://datahub.io/sebastien.lavoie/unicode-emojis/v/5 worked while /v/3, /v/4 and /v/6 didn't.

Also, https://datahub.io/core/unicode-emojis/v/194 presented an additional error (appearing twice in the output on DataHub), for which the code can be found here https://github.com/datopian/assembler/blob/master/datapackage_pipelines_assembler/processors/dump_to_s3.py#L27 :

assembler.dump_to_s3

Traceback (most recent call last):

File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/dump_to_s3.py", line 32, in <module>

MyS3Dumper()()

File "/usr/lib/python3.6/site-packages/datapackage_pipelines/lib/dump/dumper_base.py", line 47, in __call__

finalizer=self.finalize

File "/usr/lib/python3.6/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 64, in spew

for res in resources_iterator:

File "/usr/lib/python3.6/site-packages/datapackage_pipelines/lib/dump/dumper_base.py", line 150, in handle_resources

self.handle_datapackage(datapackage, parameters, stats)

File "/usr/lib/python3.6/site-packages/datapackage_pipelines/lib/dump/dumper_base.py", line 212, in handle_datapackage

location = self.write_file_to_output(temp_file_name, 'datapackage.json')

File "/usr/lib/python3.6/site-packages/datapackage_pipelines_aws/s3_dumper.py", line 46, in write_file_to_output

Key=key)

File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/dump_to_s3.py", line 27, in put_object

datetime.datetime.now()

File "/usr/lib/python3.6/site-packages/filemanager/models.py", line 103, in add_file

assert sf.owner == owner

AssertionError

After having failed for about 2 hours straight, data push was successful here for a personal account https://datahub.io/sebastien.lavoie/unicode-emojis/v/9 as well as here https://datahub.io/core/unicode-emojis for an organization account.

At this moment, the issue seems to be no longer reproducible.

rufuspollock commented 8 months ago

INVALID. This relates to datahub-v2 which is now deprecated.