asttokens 2.4.0 pyodide package missing from S3 bucket

[walski]

The latest Python 3 requirements list asttokens in version 2.4.0 (code).

When you try to run make -C ./sandbox/pyodide setup

It will run the fetch_packages make task (code)

Which relies on the requirements3.txt to download pyodide versions of these packages.

Unfortunately the metadata for asttokens package in 2.4.0 are missing. They should be here:

https://s3.amazonaws.com/grist-pynbox/pyodide/packages/v2/asttokens-2.4.0.json

This leads to a broken install where the pyodide sandbox does not work.

Can you upload that package to your bucket?

[paulfitz]

Ooh, thanks for reporting this @walski. Looking into it.

[paulfitz]

Hi @walski can you retry with the latest version of Grist on main? Unfortunately I had to make a source code change.

[walski]

@paulfitz now there are a LOT of packages for which there is no metada in S3:

make fetch_packages
make[1]: Entering directory '/app/code/sandbox/pyodide'
node ./packages.js https://s3.amazonaws.com/grist-pynbox/pyodide/packages/v3/ _build/packages/
Caching packages on disk {
  src: 'https://s3.amazonaws.com/grist-pynbox/pyodide/packages/v3/',
  dest: '_build/packages/'
}
Cached { libs: [] }
Fetching { name: 'astroid', standardName: 'astroid', version: '2.14.2' }
No metadata available {
  lib: { name: 'astroid', standardName: 'astroid', version: '2.14.2' }
}
Fetching { name: 'asttokens', standardName: 'asttokens', version: '2.4.0' }
No metadata available {
  lib: { name: 'asttokens', standardName: 'asttokens', version: '2.4.0' }
}
Fetching { name: 'chardet', standardName: 'chardet', version: '5.1.0' }
No metadata available { lib: { name: 'chardet', standardName: 'chardet', version: '5.1.0' } }
Fetching { name: 'et-xmlfile', standardName: 'et_xmlfile', version: '1.0.1' }
No metadata available {
  lib: { name: 'et-xmlfile', standardName: 'et_xmlfile', version: '1.0.1' }
}
Fetching { name: 'executing', standardName: 'executing', version: '1.1.1' }
No metadata available {
  lib: { name: 'executing', standardName: 'executing', version: '1.1.1' }
}
Fetching {
  name: 'friendly-traceback',
  standardName: 'friendly_traceback',
  version: '0.7.48'
}
No metadata available {
  lib: {
    name: 'friendly-traceback',
    standardName: 'friendly_traceback',
    version: '0.7.48'
  }
}
Fetching { name: 'iso8601', standardName: 'iso8601', version: '0.1.12' }
Fetching {
  name: 'lazy-object-proxy',
  standardName: 'lazy_object_proxy',
  version: '1.6.0'
}
No metadata available {
  lib: { name: 'iso8601', standardName: 'iso8601', version: '0.1.12' }
}
No metadata available {
  lib: {
    name: 'lazy-object-proxy',
    standardName: 'lazy_object_proxy',
    version: '1.6.0'
  }
}
Fetching { name: 'openpyxl', standardName: 'openpyxl', version: '3.0.10' }
No metadata available {
  lib: { name: 'openpyxl', standardName: 'openpyxl', version: '3.0.10' }
}
Fetching {
  name: 'phonenumberslite',
  standardName: 'phonenumberslite',
  version: '8.12.57'
}
No metadata available {
  lib: {
    name: 'phonenumberslite',
    standardName: 'phonenumberslite',
    version: '8.12.57'
  }
}
Fetching { name: 'pure-eval', standardName: 'pure_eval', version: '0.2.2' }
No metadata available {
  lib: { name: 'pure-eval', standardName: 'pure_eval', version: '0.2.2' }
}
Fetching {
  name: 'python-dateutil',
  standardName: 'python_dateutil',
  version: '2.8.2'
}
No metadata available {
  lib: {
    name: 'python-dateutil',
    standardName: 'python_dateutil',
    version: '2.8.2'
  }
}
Fetching { name: 'roman', standardName: 'roman', version: '3.3' }
No metadata available { lib: { name: 'roman', standardName: 'roman', version: '3.3' } }
Fetching { name: 'six', standardName: 'six', version: '1.16.0' }
No metadata available { lib: { name: 'six', standardName: 'six', version: '1.16.0' } }
Fetching {
  name: 'sortedcontainers',
  standardName: 'sortedcontainers',
  version: '2.4.0'
}
Fetching { name: 'stack-data', standardName: 'stack_data', version: '0.5.1' }
No metadata available {
  lib: {
    name: 'sortedcontainers',
    standardName: 'sortedcontainers',
    version: '2.4.0'
  }
}
No metadata available {
  lib: { name: 'stack-data', standardName: 'stack_data', version: '0.5.1' }
}
Fetching {
  name: 'typing-extensions',
  standardName: 'typing_extensions',
  version: '4.4.0'
}
Fetching {
  name: 'unittest-xml-reporting',
  standardName: 'unittest_xml_reporting',
  version: '2.0.0'
}
No metadata available {
  lib: {
    name: 'typing-extensions',
    standardName: 'typing_extensions',
    version: '4.4.0'
  }
}
Fetching { name: 'wrapt', standardName: 'wrapt', version: '1.15.0' }
No metadata available {
  lib: {
    name: 'unittest-xml-reporting',
    standardName: 'unittest_xml_reporting',
    version: '2.0.0'
  }
}
No metadata available { lib: { name: 'wrapt', standardName: 'wrapt', version: '1.15.0' } }
Missing {
  libs: [
    'astroid',
    'asttokens',
    'chardet',
    'et-xmlfile',
    'executing',
    'friendly-traceback',
    'iso8601',
    'lazy-object-proxy',
    'openpyxl',
    'phonenumberslite',
    'pure-eval',
    'python-dateutil',
    'roman',
    'six',
    'sortedcontainers',
    'stack-data',
    'typing-extensions',
    'unittest-xml-reporting',
    'wrapt'
  ]
}
make[1]: Leaving directory '/app/code/sandbox/pyodide'
make: Leaving directory '/app/code/sandbox/pyodide'

[paulfitz]

Missed an important step, sorry! How about now?

[walski]

@paulfitz this looks a lot better:

make fetch_packages
make[1]: Entering directory '/app/code/sandbox/pyodide'
node ./packages.js https://s3.amazonaws.com/grist-pynbox/pyodide/packages/v3/ _build/packages/
Caching packages on disk {
  src: 'https://s3.amazonaws.com/grist-pynbox/pyodide/packages/v3/',
  dest: '_build/packages/'
}
Cached { libs: [] }
Fetching { name: 'astroid', standardName: 'astroid', version: '2.14.2' }
Fetching { name: 'asttokens', standardName: 'asttokens', version: '2.4.0' }
Fetching { name: 'chardet', standardName: 'chardet', version: '5.1.0' }
Fetching { name: 'et-xmlfile', standardName: 'et_xmlfile', version: '1.0.1' }
Fetching { name: 'executing', standardName: 'executing', version: '1.1.1' }
Fetching {
  name: 'friendly-traceback',
  standardName: 'friendly_traceback',
  version: '0.7.48'
}
Fetching { name: 'iso8601', standardName: 'iso8601', version: '0.1.12' }
Fetching {
  name: 'lazy-object-proxy',
  standardName: 'lazy_object_proxy',
  version: '1.6.0'
}
Fetching { name: 'openpyxl', standardName: 'openpyxl', version: '3.0.10' }
Fetching {
  name: 'phonenumberslite',
  standardName: 'phonenumberslite',
  version: '8.12.57'
}
Fetching { name: 'pure-eval', standardName: 'pure_eval', version: '0.2.2' }
Fetching {
  name: 'python-dateutil',
  standardName: 'python_dateutil',
  version: '2.8.2'
}
Fetching { name: 'roman', standardName: 'roman', version: '3.3' }
Fetching { name: 'six', standardName: 'six', version: '1.16.0' }
Fetching {
  name: 'sortedcontainers',
  standardName: 'sortedcontainers',
  version: '2.4.0'
}
Fetching { name: 'stack-data', standardName: 'stack_data', version: '0.5.1' }
Fetching {
  name: 'typing-extensions',
  standardName: 'typing_extensions',
  version: '4.4.0'
}
Fetching {
  name: 'unittest-xml-reporting',
  standardName: 'unittest_xml_reporting',
  version: '2.0.0'
}
Fetching { name: 'wrapt', standardName: 'wrapt', version: '1.15.0' }
Missing { libs: [] }
make[1]: Leaving directory '/app/code/sandbox/pyodide'
make: Leaving directory '/app/code/sandbox/pyodide'

Works also just fine in the app 🥳

Thanks for the quick fix! Can you communicate any timeline for a new release including this as well as the (exciting!) OIDC work?

[paulfitz]

Thanks for testing and for confirming it works!

grist-core numbered releases typically happen towards the start of each month, so the next one of those would be in early December. Is that the kind of release you were asking about?

The latest tag for our docker image is updated daily, but I didn't think the docker image has pyodide packages in it (it supports gvisor sandboxing).