API: Establish `pydata_backend`

[mtsokol]

Replaces #645

Hi @hameerabbasi,

Including the new backend in pydata/sparse ended up a bit convoluted, therefore I decided to break this task into smaller steps.

Here's the first step which moves existing implementation to sparse/pydata_backend directory with tests and adds context manager that tracks backend contextvar. The test suite has a lot of pydata specific calls so it will be a separate effort to make tests common for both backends.

I applied all your comments from #645 here.

The next PR will run some stub tests with finch to see that it can be installed and run in CI.

[github-actions[bot]]

Test Results

5 899 tests  +2   5 865 :white_check_mark:  - 1   3m 18s :stopwatch: -1s     1 suites ±0      34 :zzz: +3      1 files   ±0       0 :x: ±0 

Results for commit 4c309c73. ± Comparison against base commit 92e80785.

This pull request removes 5897 and adds 5899 tests. Note that renamed tests count towards both.

``` sparse.tests.test_array_function ‑ test_binary[dot-arg_order0] sparse.tests.test_array_function ‑ test_binary[dot-arg_order1] sparse.tests.test_array_function ‑ test_binary[dot-arg_order2] sparse.tests.test_array_function ‑ test_binary[matmul-arg_order0] sparse.tests.test_array_function ‑ test_binary[matmul-arg_order1] sparse.tests.test_array_function ‑ test_binary[matmul-arg_order2] sparse.tests.test_array_function ‑ test_binary[result_type-arg_order0] sparse.tests.test_array_function ‑ test_binary[result_type-arg_order1] sparse.tests.test_array_function ‑ test_binary[result_type-arg_order2] sparse.tests.test_array_function ‑ test_binary[tensordot-arg_order0] … ``` ``` sparse.pydata_backend.tests.test_array_function ‑ test_binary[dot-arg_order0] sparse.pydata_backend.tests.test_array_function ‑ test_binary[dot-arg_order1] sparse.pydata_backend.tests.test_array_function ‑ test_binary[dot-arg_order2] sparse.pydata_backend.tests.test_array_function ‑ test_binary[matmul-arg_order0] sparse.pydata_backend.tests.test_array_function ‑ test_binary[matmul-arg_order1] sparse.pydata_backend.tests.test_array_function ‑ test_binary[matmul-arg_order2] sparse.pydata_backend.tests.test_array_function ‑ test_binary[result_type-arg_order0] sparse.pydata_backend.tests.test_array_function ‑ test_binary[result_type-arg_order1] sparse.pydata_backend.tests.test_array_function ‑ test_binary[result_type-arg_order2] sparse.pydata_backend.tests.test_array_function ‑ test_binary[tensordot-arg_order0] … ```

This pull request removes 31 skipped tests and adds 34 skipped tests. Note that renamed tests count towards both.

``` sparse.tests.test_compressed ‑ test_reductions_float16[f4-axis4-sum-kwargs0] sparse.tests.test_compressed ‑ test_reductions_float16[f8-axis4-sum-kwargs0] sparse.tests.test_compressed ‑ test_reductions_float16[i4-0-mean-kwargs1] sparse.tests.test_compressed ‑ test_reductions_float16[i4-0-sum-kwargs0] sparse.tests.test_compressed ‑ test_reductions_float16[i4-1-mean-kwargs1] sparse.tests.test_compressed ‑ test_reductions_float16[i4-1-sum-kwargs0] sparse.tests.test_compressed ‑ test_reductions_float16[i4-axis4-mean-kwargs1] sparse.tests.test_compressed ‑ test_reductions_float16[i4-axis4-sum-kwargs0] sparse.tests.test_compressed ‑ test_reductions_float16[i8-0-mean-kwargs1] sparse.tests.test_compressed ‑ test_reductions_float16[i8-0-sum-kwargs0] … ``` ``` sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[f4-None-sum-kwargs0] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[f4-axis4-sum-kwargs0] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[f8-axis4-sum-kwargs0] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-0-mean-kwargs1] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-0-sum-kwargs0] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-1-mean-kwargs1] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-1-sum-kwargs0] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-axis4-mean-kwargs1] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-axis4-sum-kwargs0] sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i8-0-mean-kwargs1] … ```

:recycle: This comment has been updated with latest results.

[hameerabbasi]

We also need a rebase.

[rgommers]

Quick question: since pydata_backend and finch_backend are (I assume) private and are only supposed to be used via context manager backend selection, wouldn't it be better to prepend underscores to the directory names?

[mtsokol]

Sure! I think they can be prepended with an underscore. There will always be backend-specific members, like GCXS only present in pydata_backend, but importing it is hidden by backend selection mechanism (env-var or context manager).