Bug: Download failed: usearch_sqlite_macos_arm64_2.10.0.dylib could not be found

[simonw]

Describe the bug

Got this after trying to upgrade to latest version to try the code from this issue again:

• 371

  python -m pip install usearch -U

  That worked:

  Collecting usearch
  Downloading usearch-2.10.0-cp310-cp310-macosx_11_0_arm64.whl (398 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 398.2/398.2 kB 2.6 MB/s eta 0:00:00
  Requirement already satisfied: numpy in /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages (from usearch) (1.26.2)
  Requirement already satisfied: tqdm in /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages (from usearch) (4.65.0)
  Installing collected packages: usearch
  Attempting uninstall: usearch
  Found existing installation: usearch 2.9.2
  Uninstalling usearch-2.9.2:
    Successfully uninstalled usearch-2.9.2
  Successfully installed usearch-2.10.0

  But then:

  Python 3.10.10 (main, Mar 21 2023, 13:41:05) [Clang 14.0.6 ] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import usearch
  /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/usearch/__init__.py:100: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)
  /opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/usearch/__init__.py:107: UserWarning: Download failed: https://github.com/unum-cloud/usearch/releases/download/v2.10.0/usearch_sqlite_macos_arm64_2.10.0.dylib could not be found.
  warnings.warn(f"Download failed: {e.url} could not be found.", UserWarning)

Steps to reproduce

Try to use import usearch against latest release on an M2 Mac.

Expected behavior

The SQLite extension should be downloaded - this should not be a 404:

https://github.com/unum-cloud/usearch/releases/download/v2.10.0/usearch_sqlite_macos_arm64_2.10.0.dylib

USearch version

2.10.0

Operating System

macOS Sonoma 14.1.1 (23B81) on 64GB Apple M2 Max

Hardware architecture

Arm

Which interface are you using?

Python bindings

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[simonw]

The error is coming from here: https://github.com/unum-cloud/usearch/blob/708fedecf516687b19df3b5f9be4bf3b1c91b824/python/usearch/__init__.py#L103-L110

Here's where that download URL is constructed:

https://github.com/unum-cloud/usearch/blob/708fedecf516687b19df3b5f9be4bf3b1c91b824/python/usearch/__init__.py#L25-L51

[ashvardanian]

I am trying to avoid a breaking change in how the SQLite module is loaded, and it really overcomplicates things. Working on a hot-fix now 🤗

[ashvardanian]

@simonw there are a couple of issues here.

1. SQLite will add .dylib suffix at the end of the name looking for a .dylib.dylib file.
2. Even after succesfull download, my M2 Mac complains about the build compatibility.

>>> conn.load_extension(usearch.sqlite.removesuffix('.dylib'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/var/folders/r0/mw8phgys4lg_60kgh8xm_x780000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib, 0x000A): tried: '/var/folders/r0/mw8phgys4lg_60kgh8xm_x780000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/var/folders/r0/mw8phgys4lg_60kgh8xm_x780000gn/T/usearch_

That doesn't happen with local builds, so it's somewhere in the CI. Still checking 😉

[ashvardanian]

:tada: This issue has been resolved in version 2.10.1 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

[ashvardanian]

I am gonna change the import method, which is technical a breaking change and should trigger a major release, but assuming the low adoption of the SQLite API, I hope it's fine 🤗

[simonw]

It's not quite working for me still:

```pycon
>>> import usearch
/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/site-packages/usearch/__init__.py:100: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)
>>> import sqlite3
>>> conn = sqlite3.connect(":memory:")
>>> conn.enable_load_extension(True)
>>> conn.load_extension(usearch.sqlite_path())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'usearch' has no attribute 'sqlite_path'
>>> usearch.sqlite
'/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib'
>>> conn.load_extension(usearch.sqlite)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib.dylib, 0x000A): tried: '/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib.dylib' (no such file), '/var
>>> conn.lead_extension(usearch.sqlite.replace('.dylib', ''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'sqlite3.Connection' object has no attribute 'lead_extension'. Did you mean: 'load_extension'?
>>> conn.load_extension(usearch.sqlite.replace('.dylib', ''))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib, 0x000A): tried: '/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_sqlite_macos_arm64_2.10.0.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/usearch_

It says it got x86_64 but needed arm64.

Also note I had to strip the .dylib from the end of the path - plus usearch.sqlite_path() has been renamed usearch.sqlite` - that's a bug in this README: https://github.com/unum-cloud/usearch/blob/3fd51841664471fad12ef5c37f0d87e783a7c7a1/sqlite/README.md#L28-L33

[simonw]

I tried downloading this: https://github.com/unum-cloud/usearch/releases/download/v2.10.1/usearch_sqlite_macos_arm64_2.10.1.dylib

Then opening it like this:

>>> conn.load_extension("/tmp/usearch_sqlite_macos_arm64_2.10.1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlopen(/tmp/usearch_sqlite_macos_arm64_2.10.1.dylib, 0x000A): tried: '/tmp/usearch_sqlite_macos_arm64_2.10.1.dylib' (code signature in <DCF25A47-045B-3D73-BE3E-6EF40C87A897> '/private/tmp/usearch_sqlite_macos_arm64_2.10.1.dylib' not valid for use in process: library load disallowed by system policy), '/System/Volumes/Preboot/Cryptexes

I fixed that error by opening up the macOS System Preferences and going to Security and allowing that binary. On a second attempt:

>>> conn.load_extension("/tmp/usearch_sqlite_macos_arm64_2.10.1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: dlsym(0x81325550, sqlite3_usearchsqlitemacosarm_init): symbol not found

[ashvardanian]

Yes, the right architecture wasn't selected in CI for the 2.10.0 build, that you are using. But the 2.10.1 should be fixing that. The CI can take a couple of hours to complete all the tests and I will try again afterwards.

---

@simonw, your log, however, is quite handy, as it highlights another issue - SQLite is searching for a weird symbol name when loading the library - sqlite3_usearchsqlitemacosarm_init. That will be an issue even with the new installation. If you git clone and then pip install -e . locally, the issue won't happen. Alternatively, you can rename the binary to usearch_sqlite.dylib locally, it may pass.

---

Give me a few hors to polish this. Sorry for inconvenience, debugging GitHub CI is an absolute nightmare 🧠 🔫

[ashvardanian]

@simonw, can you please check the 2.10.3? It should work fine.

[simonw]

That worked perfectly!

$ pip install usearch
Collecting usearch
  Downloading usearch-2.10.3-cp311-cp311-macosx_11_0_arm64.whl (400 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 400.3/400.3 kB 2.8 MB/s eta 0:00:00
Requirement already satisfied: numpy in /Users/simon/.local/share/virtualenvs/foop-q39PfZK-/lib/python3.11/site-packages (from usearch) (1.24.3)
Requirement already satisfied: tqdm in /Users/simon/.local/share/virtualenvs/foop-q39PfZK-/lib/python3.11/site-packages (from usearch) (4.65.0)
Installing collected packages: usearch
Successfully installed usearch-2.10.3
$ python

Python 3.11.8 (main, Feb  6 2024, 21:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import usearch
>>> usearch.sqlite_path()
/Users/simon/.local/share/virtualenvs/foop-q39PfZK-/lib/python3.11/site-packages/usearch/__init__.py:93: UserWarning: Will download `usearch_sqlite` binary from GitHub.
  warnings.warn("Will download `usearch_sqlite` binary from GitHub.", UserWarning)
'/Users/simon/.local/share/virtualenvs/foop-q39PfZK-/bin/usearch_binaries/usearch_sqlite'
>>> import sqlite3
>>> conn = sqlite3.connect(":memory:")
>>> conn.enable_load_extension(True)
>>> conn.load_extension(usearch.sqlite_path())
>>> [fn[0] for fn in conn.execute('pragma function_list').fetchall()]
['pow', 'group_concat', 'group_concat', 'json_type', 'json_type', 'jsonb_set', 'julianday', 'ntile', 'nullif', 'sqlite_compileoption_get', 'json_valid', 'json_valid', 'json_quote', 'json_patch', '->', 'json_array', 'current_timestamp', 'power', 'sqlite_compileoption_used', 'json_remove', 'jsonb_patch', 'json_object', 'json_insert', '->>', 'jsonb_array', 'sin', 'sum', 'quote', 'printf', 'likelihood', 'json_replace', 'jsonb_remove', 'jsonb_object', 'jsonb_insert', 'json_extract', 'last_value', 'rank', 'sign', 'sqrt', 'sinh', 'tan', 'round', 'round', 'rtrim', 'rtrim', 'jsonb_replace', 'jsonb_extract', 'nth_value', 'tanh', 'random', 'trim', 'trim', 'time', 'radians', 'trunc', 'total', 'substr', 'substr', 'replace', 'unhex', 'unhex', 'upper', 'subtype', 'typeof', 'load_extension', 'load_extension', 'json_group_array', 'avg', 'abs', 'octet_length', 'json_group_object', 'jsonb_group_array', 'json_array_length', 'json_array_length', 'strftime', 'atan', 'asin', 'acos', 'substring', 'substring', 'randomblob', 'unicode', 'jsonb_group_object', 'timediff', 'percent_rank', 'row_number', 'atanh', 'asinh', 'acosh', 'cos', 'atan2', 'string_agg', 'last_insert_rowid', 'sqlite_log', 'unlikely', 'json_error_position', 'cosh', 'ceil', 'char', 'unixepoch', 'exp', 'count', 'count', 'date', 'concat', 'ceiling', 'total_changes', 'changes', 'sqlite_version', 'degrees', 'floor', 'coalesce', 'glob', 'zeroblob', 'hex', 'iif', 'sqlite_source_id', 'concat_ws', 'format', 'datetime', 'cume_dist', 'ln', 'instr', 'json', 'dense_rank', 'log', 'log', 'ifnull', 'jsonb', 'current_date', 'current_time', 'lag', 'lag', 'lag', 'mod', 'log2', 'like', 'like', 'max', 'max', 'min', 'min', 'lead', 'lead', 'lead', 'log10', 'lower', 'ltrim', 'ltrim', 'first_value', 'pi', 'length', 'likely', 'json_set', 'distance_inner_i8', 'distance_cosine_i8', 'distance_inner_f16', 'distance_sqeuclidean_f16', 'distance_divergence_f32', 'fts3_tokenizer', 'fts3_tokenizer', 'distance_divergence_i8', 'bm25', 'distance_jaccard_binary', 'distance_sqeuclidean_i8', 'highlight', 'rtreenode', 'rtreecheck', 'geopoly_regular', 'matchinfo', 'matchinfo', 'geopoly_debug', 'geopoly_bbox', 'distance_hamming_unicode', 'fts5_source_id', 'geopoly_contains_point', 'geopoly_svg', 'distance_haversine_meters', 'snippet', 'geopoly_overlap', 'distance_sqeuclidean_f64', 'distance_hamming_bytes', 'distance_levenshtein_unicode', 'distance_cosine_f32', 'distance_cosine_f16', 'rtreedepth', 'geopoly_area', 'match', 'geopoly_blob', 'optimize', 'geopoly_within', 'distance_inner_f64', 'geopoly_json', 'offsets', 'distance_cosine_f64', 'geopoly_xform', 'distance_divergence_f16', 'fts5', 'distance_sqeuclidean_f32', 'geopoly_ccw', 'distance_hamming_binary', 'distance_levenshtein_bytes', 'distance_divergence_f64', 'geopoly_group_bbox', 'distance_inner_f32']
>>> conn.execute('select distance_cosine_f32(?, ?)', (repr([1.1, 2, 3]), repr([3.1, 1, 2]))).fetchall()
[(0.20811247825622559,)]