data-apis / array-api-tests

Test suite for Python array API standard compliance
https://data-apis.org/array-api-tests/
MIT License
65 stars 44 forks source link

signature tests being misled by **kwargs #171

Open asmeurer opened 1 year ago

asmeurer commented 1 year ago

The meshgrid signature test fails with cupy:

E           AssertionError: Argument 'indexing' missing from signature
E           assert 'indexing' in odict_keys(['xi', 'kwargs'])
E            +  where 'indexing' = <Parameter "indexing: str = 'xy'">.name
E            +  and   odict_keys(['xi', 'kwargs']) = <built-in method keys of mappingproxy object at 0x7fc946212170>()
E            +    where <built-in method keys of mappingproxy object at 0x7fc946212170> = mappingproxy(OrderedDict([('xi', <Parameter "*xi">), ('kwargs', <Parameter "**kwargs">)])).keys
E            +      where mappingproxy(OrderedDict([('xi', <Parameter "*xi">), ('kwargs', <Parameter "**kwargs">)])) = <Signature (*xi, **kwargs)>.parameters

array_api_tests/test_signatures.py:92: AssertionError

cupy.meshgrid does have the indexing keyword, but its signature is meshgrid(*x, **kwargs). If we can't check a keyword argument directly, we should assume it is present with **kwargs, and let the individual tests suss out whether it really supports it.

honno commented 1 year ago

Interesting scenario!

cupy.meshgrid does have the indexing keyword, but its signature is meshgrid(*x, **kwargs). If we can't check a keyword argument directly, we should assume it is present with **kwargs, and let the individual tests suss out whether it really supports it.

Personally I'd rather skip here (after testing everything else first so the test still could fail), as we can't confidently say the test "passes", with a skip reason like "keyword not found, but star-kwargs could consume keyword".