search

pydata / patsy

Describing statistical models in Python using symbolic formulas
Other
955 stars 104 forks source link

Test failures with Python 3.12 #196

Closed mgorny closed 12 months ago

mgorny commented 1 year ago

Tested on 0.5.3 and 6bd31ca7f5f12e77a918681d14b7c28d931bde38.

$ tox -e py312
py312: install_deps> python -I -m pip install '.[test]'
.pkg: install_requires> python -I -m pip install 'setuptools>=40.8.0' wheel
.pkg: _optional_hooks> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: install_requires_for_build_wheel> python -I -m pip install wheel
.pkg: prepare_metadata_for_build_wheel> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
py312: install_package_deps> python -I -m pip install 'numpy>=1.4' six
py312: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/patsy/.tox/.tmp/package/1/patsy-0.5.3+dev.tar.gz
py312: commands[0]> pytest -vv --cov=patsy --cov-config=/tmp/patsy/.coveragerc --cov-report=term-missing --cov-report=xml --cov-report=html:/tmp/patsy/.tox/coverage/py312
========================================================= test session starts =========================================================
platform linux -- Python 3.12.0, pytest-7.4.3, pluggy-1.3.0 -- /tmp/patsy/.tox/py312/bin/python
cachedir: .tox/py312/.pytest_cache
rootdir: /tmp/patsy
configfile: setup.cfg
testpaths: patsy
plugins: cov-4.1.0
collected 148 items                                                                                                                   

patsy/build.py::test__max_allowed_dim PASSED                                                                                    [  0%]
patsy/build.py::test__eval_factor_numerical PASSED                                                                              [  1%]
patsy/build.py::test__eval_factor_categorical PASSED                                                                            [  2%]
patsy/build.py::test__column_combinations PASSED                                                                                [  2%]
patsy/build.py::test__subterm_column_names_iter_and__build_subterm PASSED                                                       [  3%]
patsy/build.py::test__factors_memorize PASSED                                                                                   [  4%]
patsy/build.py::test__examine_factor_types PASSED                                                                               [  4%]
patsy/builtins.py::test_I PASSED                                                                                                [  5%]
patsy/builtins.py::test_Q PASSED                                                                                                [  6%]
patsy/categorical.py::test_C PASSED                                                                                             [  6%]
patsy/categorical.py::test_guess_categorical PASSED                                                                             [  7%]
patsy/categorical.py::test_CategoricalSniffer PASSED                                                                            [  8%]
patsy/categorical.py::test_categorical_to_int PASSED                                                                            [  8%]
patsy/constraint.py::test_LinearConstraint PASSED                                                                               [  9%]
patsy/constraint.py::test_LinearConstraint_combine PASSED                                                                       [ 10%]
patsy/constraint.py::test__tokenize_constraint PASSED                                                                           [ 10%]
patsy/constraint.py::test_linear_constraint PASSED                                                                              [ 11%]
patsy/constraint.py::test_eval_errors PASSED                                                                                    [ 12%]
patsy/contrasts.py::test_ContrastMatrix PASSED                                                                                  [ 12%]
patsy/contrasts.py::test__obj_to_readable_str PASSED                                                                            [ 13%]
patsy/contrasts.py::test__name_levels PASSED                                                                                    [ 14%]
patsy/contrasts.py::test__get_level PASSED                                                                                      [ 14%]
patsy/contrasts.py::test_Treatment PASSED                                                                                       [ 15%]
patsy/contrasts.py::test_Poly PASSED                                                                                            [ 16%]
patsy/contrasts.py::test_Sum PASSED                                                                                             [ 16%]
patsy/contrasts.py::test_Helmert PASSED                                                                                         [ 17%]
patsy/contrasts.py::test_diff PASSED                                                                                            [ 18%]
patsy/desc.py::test_Term PASSED                                                                                                 [ 18%]
patsy/desc.py::test_ModelDesc PASSED                                                                                            [ 19%]
patsy/desc.py::test_ModelDesc_from_formula PASSED                                                                               [ 20%]
patsy/desc.py::test_eval_formula PASSED                                                                                         [ 20%]
patsy/desc.py::test_eval_formula_error_reporting FAILED                                                                         [ 21%]
patsy/desc.py::test_formula_factor_origin PASSED                                                                                [ 22%]
patsy/design_info.py::test_FactorInfo PASSED                                                                                    [ 22%]
patsy/design_info.py::test_SubtermInfo PASSED                                                                                   [ 23%]
patsy/design_info.py::test_DesignInfo PASSED                                                                                    [ 24%]
patsy/design_info.py::test_DesignInfo_from_array PASSED                                                                         [ 25%]
patsy/design_info.py::test_DesignInfo_linear_constraint PASSED                                                                  [ 25%]
patsy/design_info.py::test_DesignInfo_deprecated_attributes PASSED                                                              [ 26%]
patsy/design_info.py::test__format_float_column PASSED                                                                          [ 27%]
patsy/design_info.py::test_design_matrix PASSED                                                                                 [ 27%]
patsy/eval.py::test_VarLookupDict PASSED                                                                                        [ 28%]
patsy/eval.py::test_ast_names PASSED                                                                                            [ 29%]
patsy/eval.py::test_ast_names_disallowed_nodes PASSED                                                                           [ 29%]
patsy/eval.py::test_EvalEnvironment_capture_namespace PASSED                                                                    [ 30%]
patsy/eval.py::test_EvalEnvironment_capture_flags PASSED                                                                        [ 31%]
patsy/eval.py::test_EvalEnvironment_eval_namespace PASSED                                                                       [ 31%]
patsy/eval.py::test_EvalEnvironment_eval_flags PASSED                                                                           [ 32%]
patsy/eval.py::test_EvalEnvironment_subset PASSED                                                                               [ 33%]
patsy/eval.py::test_EvalEnvironment_eq PASSED                                                                                   [ 33%]
patsy/eval.py::test_EvalFactor_basics PASSED                                                                                    [ 34%]
patsy/eval.py::test_EvalFactor_memorize_passes_needed PASSED                                                                    [ 35%]
patsy/eval.py::test_EvalFactor_end_to_end PASSED                                                                                [ 35%]
patsy/eval.py::test_annotated_tokens PASSED                                                                                     [ 36%]
patsy/eval.py::test_replace_bare_funcalls PASSED                                                                                [ 37%]
patsy/eval.py::test_capture_obj_method_calls PASSED                                                                             [ 37%]
patsy/infix_parser.py::test_infix_parse PASSED                                                                                  [ 38%]
patsy/mgcv_cubic_splines.py::test__map_cyclic PASSED                                                                            [ 39%]
patsy/mgcv_cubic_splines.py::test__map_cyclic_errors PASSED                                                                     [ 39%]
patsy/mgcv_cubic_splines.py::test__row_tensor_product_errors PASSED                                                             [ 40%]
patsy/mgcv_cubic_splines.py::test__row_tensor_product PASSED                                                                    [ 41%]
patsy/mgcv_cubic_splines.py::test__get_all_sorted_knots PASSED                                                                  [ 41%]
patsy/mgcv_cubic_splines.py::test_crs_errors PASSED                                                                             [ 42%]
patsy/mgcv_cubic_splines.py::test_crs_compat PASSED                                                                             [ 43%]
patsy/mgcv_cubic_splines.py::test_crs_with_specific_constraint PASSED                                                           [ 43%]
patsy/mgcv_cubic_splines.py::test_te_errors PASSED                                                                              [ 44%]
patsy/mgcv_cubic_splines.py::test_te_1smooth PASSED                                                                             [ 45%]
patsy/mgcv_cubic_splines.py::test_te_2smooths PASSED                                                                            [ 45%]
patsy/mgcv_cubic_splines.py::test_te_3smooths PASSED                                                                            [ 46%]
patsy/missing.py::test_NAAction_basic PASSED                                                                                    [ 47%]
patsy/missing.py::test_NAAction_NA_types_numerical PASSED                                                                       [ 47%]
patsy/missing.py::test_NAAction_NA_types_categorical PASSED                                                                     [ 48%]
patsy/missing.py::test_NAAction_drop PASSED                                                                                     [ 49%]
patsy/missing.py::test_NAAction_raise PASSED                                                                                    [ 50%]
patsy/origin.py::test_Origin PASSED                                                                                             [ 50%]
patsy/parse_formula.py::test__tokenize_formula PASSED                                                                           [ 51%]
patsy/parse_formula.py::test_parse_formula PASSED                                                                               [ 52%]
patsy/parse_formula.py::test_parse_origin PASSED                                                                                [ 52%]
patsy/parse_formula.py::test_parse_errors FAILED                                                                                [ 53%]
patsy/parse_formula.py::test_parse_extra_op FAILED                                                                              [ 54%]
patsy/redundancy.py::test__Subterm PASSED                                                                                       [ 54%]
patsy/redundancy.py::test__subsets_sorted PASSED                                                                                [ 55%]
patsy/redundancy.py::test__simplify_subterms PASSED                                                                             [ 56%]
patsy/redundancy.py::test_pick_contrasts_for_term PASSED                                                                        [ 56%]
patsy/splines.py::test__R_compat_quantile PASSED                                                                                [ 57%]
patsy/splines.py::test_bs_compat PASSED                                                                                         [ 58%]
patsy/splines.py::test_bs_0degree PASSED                                                                                        [ 58%]
patsy/splines.py::test_bs_errors PASSED                                                                                         [ 59%]
patsy/test_build.py::test_assert_full_rank PASSED                                                                               [ 60%]
patsy/test_build.py::test_simple PASSED                                                                                         [ 60%]
patsy/test_build.py::test_R_bugs PASSED                                                                                         [ 61%]
patsy/test_build.py::test_redundancy_thoroughly PASSED                                                                          [ 62%]
patsy/test_build.py::test_data_types PASSED                                                                                     [ 62%]
patsy/test_build.py::test_build_design_matrices_dtype PASSED                                                                    [ 63%]
patsy/test_build.py::test_return_type PASSED                                                                                    [ 64%]
patsy/test_build.py::test_NA_action PASSED                                                                                      [ 64%]
patsy/test_build.py::test_NA_drop_preserves_levels PASSED                                                                       [ 65%]
patsy/test_build.py::test_return_type_pandas PASSED                                                                             [ 66%]
patsy/test_build.py::test_data_mismatch PASSED                                                                                  [ 66%]
patsy/test_build.py::test_data_independent_builder PASSED                                                                       [ 67%]
patsy/test_build.py::test_same_factor_in_two_matrices PASSED                                                                    [ 68%]
patsy/test_build.py::test_eval_env_type_builder PASSED                                                                          [ 68%]
patsy/test_build.py::test_categorical PASSED                                                                                    [ 69%]
patsy/test_build.py::test_contrast PASSED                                                                                       [ 70%]
patsy/test_build.py::test_DesignInfo_subset PASSED                                                                              [ 70%]
patsy/test_highlevel.py::test_formula_likes PASSED                                                                              [ 71%]
patsy/test_highlevel.py::test_return_pandas PASSED                                                                              [ 72%]
patsy/test_highlevel.py::test_term_info PASSED                                                                                  [ 72%]
patsy/test_highlevel.py::test_data_types PASSED                                                                                 [ 73%]
patsy/test_highlevel.py::test_categorical PASSED                                                                                [ 74%]
patsy/test_highlevel.py::test_builtins PASSED                                                                                   [ 75%]
patsy/test_highlevel.py::test_incremental PASSED                                                                                [ 75%]
patsy/test_highlevel.py::test_env_transform PASSED                                                                              [ 76%]
patsy/test_highlevel.py::test_term_order PASSED                                                                                 [ 77%]
patsy/test_highlevel.py::test_future PASSED                                                                                     [ 77%]
patsy/test_highlevel.py::test_multicolumn PASSED                                                                                [ 78%]
patsy/test_highlevel.py::test_dmatrix_dmatrices_no_data PASSED                                                                  [ 79%]
patsy/test_highlevel.py::test_designinfo_describe PASSED                                                                        [ 79%]
patsy/test_highlevel.py::test_evalfactor_reraise PASSED                                                                         [ 80%]
patsy/test_highlevel.py::test_dmatrix_NA_action PASSED                                                                          [ 81%]
patsy/test_highlevel.py::test_0d_data PASSED                                                                                    [ 81%]
patsy/test_highlevel.py::test_env_not_saved_in_builder PASSED                                                                   [ 82%]
patsy/test_highlevel.py::test_C_and_pandas_categorical PASSED                                                                   [ 83%]
patsy/test_regressions.py::test_issue_11 PASSED                                                                                 [ 83%]
patsy/test_state.py::test_Center PASSED                                                                                         [ 84%]
patsy/test_state.py::test_stateful_transform_wrapper PASSED                                                                     [ 85%]
patsy/test_state.py::test_Standardize PASSED                                                                                    [ 85%]
patsy/tokens.py::test_python_tokenize FAILED                                                                                    [ 86%]
patsy/tokens.py::test_pretty_untokenize_and_normalize_token_spacing PASSED                                                      [ 87%]
patsy/user_util.py::test_balanced PASSED                                                                                        [ 87%]
patsy/user_util.py::test_demo_data PASSED                                                                                       [ 88%]
patsy/user_util.py::test_LookupFactor PASSED                                                                                    [ 89%]
patsy/util.py::test_asarray_or_pandas PASSED                                                                                    [ 89%]
patsy/util.py::test_atleast_2d_column_default PASSED                                                                            [ 90%]
patsy/util.py::test_pandas_friendly_reshape PASSED                                                                              [ 91%]
patsy/util.py::test_to_uniqueify_list PASSED                                                                                    [ 91%]
patsy/util.py::test_wide_dtype_for_and_widen PASSED                                                                             [ 92%]
patsy/util.py::test_PushbackAdapter PASSED                                                                                      [ 93%]
patsy/util.py::test_repr_pretty PASSED                                                                                          [ 93%]
patsy/util.py::test_SortAnythingKey PASSED                                                                                      [ 94%]
patsy/util.py::test_safe_scalar_isnan PASSED                                                                                    [ 95%]
patsy/util.py::test_safe_isnan PASSED                                                                                           [ 95%]
patsy/util.py::test_iterable PASSED                                                                                             [ 96%]
patsy/util.py::test_pandas_Categorical_from_codes PASSED                                                                        [ 97%]
patsy/util.py::test_pandas_Categorical_accessors PASSED                                                                         [ 97%]
patsy/util.py::test_safe_is_pandas_categorical PASSED                                                                           [ 98%]
patsy/util.py::test_safe_issubdtype PASSED                                                                                      [ 99%]
patsy/util.py::test_safe_string_eq PASSED                                                                                       [100%]

============================================================== FAILURES ===============================================================
__________________________________________________ test_eval_formula_error_reporting __________________________________________________

    def test_eval_formula_error_reporting():
        from patsy.parse_formula import _parsing_error_test
        parse_fn = lambda formula: ModelDesc.from_formula(formula)
>       _parsing_error_test(parse_fn, _eval_error_tests)

patsy/desc.py:617: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
patsy/parse_formula.py:273: in _parsing_error_test
    parse_fn(bad_code)
patsy/desc.py:616: in <lambda>
    parse_fn = lambda formula: ModelDesc.from_formula(formula)
patsy/desc.py:164: in from_formula
    tree = parse_formula(tree_or_string)
patsy/parse_formula.py:146: in parse_formula
    tree = infix_parse(_tokenize_formula(code, operator_strings),
patsy/infix_parser.py:210: in infix_parse
    for token in token_source:
patsy/parse_formula.py:89: in _tokenize_formula
    for pytype, token_string, origin in it:
patsy/util.py:349: in next
    return six.advance_iterator(self._it)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

code = 'a + ('

    def python_tokenize(code):
        # Since formulas can only contain Python expressions, and Python
        # expressions cannot meaningfully contain newlines, we'll just remove all
        # the newlines up front to avoid any complications:
        code = code.replace("\n", " ").strip()
        it = tokenize.generate_tokens(StringIO(code).readline)
        try:
            for (pytype, string, (_, start), (_, end), code) in it:
                if pytype == tokenize.ENDMARKER:
                    break
                origin = Origin(code, start, end)
>               assert pytype != tokenize.NL
E               assert 65 != 65
E                +  where 65 = tokenize.NL

patsy/tokens.py:35: AssertionError
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
a <+>
'a +' 2 3
expected a noun, but instead the expression ended
    a +
      ^
a + <(>
'a + (' 4 5
__________________________________________________________ test_parse_errors __________________________________________________________

extra_operators = []

    def test_parse_errors(extra_operators=[]):
        def parse_fn(code):
            return parse_formula(code, extra_operators=extra_operators)
>       _parsing_error_test(parse_fn, _parser_error_tests)

patsy/parse_formula.py:285: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
patsy/parse_formula.py:273: in _parsing_error_test
    parse_fn(bad_code)
patsy/parse_formula.py:284: in parse_fn
    return parse_formula(code, extra_operators=extra_operators)
patsy/parse_formula.py:146: in parse_formula
    tree = infix_parse(_tokenize_formula(code, operator_strings),
patsy/infix_parser.py:210: in infix_parse
    for token in token_source:
patsy/parse_formula.py:89: in _tokenize_formula
    for pytype, token_string, origin in it:
patsy/util.py:349: in next
    return six.advance_iterator(self._it)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

code = 'a + ('

    def python_tokenize(code):
        # Since formulas can only contain Python expressions, and Python
        # expressions cannot meaningfully contain newlines, we'll just remove all
        # the newlines up front to avoid any complications:
        code = code.replace("\n", " ").strip()
        it = tokenize.generate_tokens(StringIO(code).readline)
        try:
            for (pytype, string, (_, start), (_, end), code) in it:
                if pytype == tokenize.ENDMARKER:
                    break
                origin = Origin(code, start, end)
>               assert pytype != tokenize.NL
E               assert 65 != 65
E                +  where 65 = tokenize.NL

patsy/tokens.py:35: AssertionError
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
a <+>
'a +' 2 3
expected a noun, but instead the expression ended
    a +
      ^
a + <(>
'a + (' 4 5
_________________________________________________________ test_parse_extra_op _________________________________________________________

    def test_parse_extra_op():
        extra_operators = [Operator("|", 2, 250)]
        _do_parse_test(_parser_tests,
                       extra_operators=extra_operators)
        _do_parse_test(_extra_op_parser_tests,
                       extra_operators=extra_operators)
>       test_parse_errors(extra_operators=extra_operators)

patsy/parse_formula.py:298: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
patsy/parse_formula.py:285: in test_parse_errors
    _parsing_error_test(parse_fn, _parser_error_tests)
patsy/parse_formula.py:273: in _parsing_error_test
    parse_fn(bad_code)
patsy/parse_formula.py:284: in parse_fn
    return parse_formula(code, extra_operators=extra_operators)
patsy/parse_formula.py:146: in parse_formula
    tree = infix_parse(_tokenize_formula(code, operator_strings),
patsy/infix_parser.py:210: in infix_parse
    for token in token_source:
patsy/parse_formula.py:89: in _tokenize_formula
    for pytype, token_string, origin in it:
patsy/util.py:349: in next
    return six.advance_iterator(self._it)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

code = 'a + ('

    def python_tokenize(code):
        # Since formulas can only contain Python expressions, and Python
        # expressions cannot meaningfully contain newlines, we'll just remove all
        # the newlines up front to avoid any complications:
        code = code.replace("\n", " ").strip()
        it = tokenize.generate_tokens(StringIO(code).readline)
        try:
            for (pytype, string, (_, start), (_, end), code) in it:
                if pytype == tokenize.ENDMARKER:
                    break
                origin = Origin(code, start, end)
>               assert pytype != tokenize.NL
E               assert 65 != 65
E                +  where 65 = tokenize.NL

patsy/tokens.py:35: AssertionError
-------------------------------------------------------- Captured stdout call ---------------------------------------------------------
'' ['~', '1']
ParseNode('~', Token('~', <Origin ->~<- 1 (0-1)>), [ParseNode('ONE', Token('ONE', <Origin ~ ->1<- (2-3)>, extra='1'), [])])
' ' ['~', '1']
ParseNode('~', Token('~', <Origin ->~<- 1 (0-1)>), [ParseNode('ONE', Token('ONE', <Origin ~ ->1<- (2-3)>, extra='1'), [])])
' \n ' ['~', '1']
ParseNode('~', Token('~', <Origin ->~<- 1 (0-1)>), [ParseNode('ONE', Token('ONE', <Origin ~ ->1<- (2-3)>, extra='1'), [])])
'1' ['~', '1']
ParseNode('~', None, [ParseNode('ONE', Token('ONE', <Origin ->1<- (0-1)>, extra='1'), [])])
'a' ['~', 'a']
ParseNode('~', None, [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- (0-1)>, extra='a'), [])])
'a ~ b' ['~', 'a', 'b']
ParseNode('~', Token('~', <Origin a ->~<- b (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- ~ b (0-1)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a ~ ->b<- (4-5)>, extra='b'), [])])
'(a ~ b)' ['~', 'a', 'b']
ParseNode('~', Token('~', <Origin (a ->~<- b) (3-4)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin (->a<- ~ b) (1-2)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin (a ~ ->b<-) (5-6)>, extra='b'), [])])
'a ~ ((((b))))' ['~', 'a', 'b']
ParseNode('~', Token('~', <Origin a ->~<- ((((b)))) (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- ~ ((((b)))) (0-1)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a ~ ((((->b<-)))) (8-9)>, extra='b'), [])])
'a ~ ((((+b))))' ['~', 'a', ['+', 'b']]
ParseNode('~', Token('~', <Origin a ->~<- ((((+b)))) (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- ~ ((((+b)))) (0-1)>, extra='a'), []), ParseNode('+', Token('+', <Origin a ~ ((((->+<-b)))) (8-9)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a ~ ((((+->b<-)))) (9-10)>, extra='b'), [])])])
'a + b + c' ['~', ['+', ['+', 'a', 'b'], 'c']]
ParseNode('~', None, [ParseNode('+', Token('+', <Origin a + b ->+<- c (6-7)>), [ParseNode('+', Token('+', <Origin a ->+<- b + c (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- + b + c (0-1)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + ->b<- + c (4-5)>, extra='b'), [])]), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + b + ->c<- (8-9)>, extra='c'), [])])])
'a + (b ~ c) + d' ['~', ['+', ['+', 'a', ['~', 'b', 'c']], 'd']]
ParseNode('~', None, [ParseNode('+', Token('+', <Origin a + (b ~ c) ->+<- d (12-13)>), [ParseNode('+', Token('+', <Origin a ->+<- (b ~ c) + d (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- + (b ~ c) + d (0-1)>, extra='a'), []), ParseNode('~', Token('~', <Origin a + (b ->~<- c) + d (7-8)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + (->b<- ~ c) + d (5-6)>, extra='b'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + (b ~ ->c<-) + d (9-10)>, extra='c'), [])])]), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + (b ~ c) + ->d<- (14-15)>, extra='d'), [])])])
'a + np.log(a, base=10)' ['~', ['+', 'a', 'np.log(a, base=10)']]
ParseNode('~', None, [ParseNode('+', Token('+', <Origin a ->+<- np.log(a, base=10) (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- + np.log(a, base=10) (0-1)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + ->np.log(a, base=10)<- (4-22)>, extra='np.log(a, base=10)'), [])])])
'a + np . log(a , base = 10)' ['~', ['+', 'a', 'np.log(a, base=10)']]
ParseNode('~', None, [ParseNode('+', Token('+', <Origin a ->+<- np . log(a , base = 10) (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- + np . log(a , base = 10) (0-1)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + ->np . log(a , base = 10)<- (4-27)>, extra='np.log(a, base=10)'), [])])])
'a + b ~ c * d' ['~', ['+', 'a', 'b'], ['*', 'c', 'd']]
ParseNode('~', Token('~', <Origin a + b ->~<- c * d (6-7)>), [ParseNode('+', Token('+', <Origin a ->+<- b ~ c * d (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- + b ~ c * d (0-1)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + ->b<- ~ c * d (4-5)>, extra='b'), [])]), ParseNode('*', Token('*', <Origin a + b ~ c ->*<- d (10-11)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + b ~ ->c<- * d (8-9)>, extra='c'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + b ~ c * ->d<- (12-13)>, extra='d'), [])])])
'a + b * c' ['~', ['+', 'a', ['*', 'b', 'c']]]
ParseNode('~', None, [ParseNode('+', Token('+', <Origin a ->+<- b * c (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- + b * c (0-1)>, extra='a'), []), ParseNode('*', Token('*', <Origin a + b ->*<- c (6-7)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + ->b<- * c (4-5)>, extra='b'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + b * ->c<- (8-9)>, extra='c'), [])])])])
'-a**2' ['~', ['-', ['**', 'a', '2']]]
ParseNode('~', None, [ParseNode('-', Token('-', <Origin ->-<-a**2 (0-1)>), [ParseNode('**', Token('**', <Origin -a->**<-2 (2-4)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin -->a<-**2 (1-2)>, extra='a'), []), ParseNode('NUMBER', Token('NUMBER', <Origin -a**->2<- (4-5)>, extra='2'), [])])])])
'-a:b' ['~', ['-', [':', 'a', 'b']]]
ParseNode('~', None, [ParseNode('-', Token('-', <Origin ->-<-a:b (0-1)>), [ParseNode(':', Token(':', <Origin -a->:<-b (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin -->a<-:b (1-2)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin -a:->b<- (3-4)>, extra='b'), [])])])])
'a + b:c' ['~', ['+', 'a', [':', 'b', 'c']]]
ParseNode('~', None, [ParseNode('+', Token('+', <Origin a ->+<- b:c (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- + b:c (0-1)>, extra='a'), []), ParseNode(':', Token(':', <Origin a + b->:<-c (5-6)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + ->b<-:c (4-5)>, extra='b'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a + b:->c<- (6-7)>, extra='c'), [])])])])
'(a + b):c' ['~', [':', ['+', 'a', 'b'], 'c']]
ParseNode('~', None, [ParseNode(':', Token(':', <Origin (a + b)->:<-c (7-8)>), [ParseNode('+', Token('+', <Origin (a ->+<- b):c (3-4)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin (->a<- + b):c (1-2)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin (a + ->b<-):c (5-6)>, extra='b'), [])]), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin (a + b):->c<- (8-9)>, extra='c'), [])])])
'a*b:c' ['~', ['*', 'a', [':', 'b', 'c']]]
ParseNode('~', None, [ParseNode('*', Token('*', <Origin a->*<-b:c (1-2)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<-*b:c (0-1)>, extra='a'), []), ParseNode(':', Token(':', <Origin a*b->:<-c (3-4)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a*->b<-:c (2-3)>, extra='b'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a*b:->c<- (4-5)>, extra='c'), [])])])])
'a+b / c' ['~', ['+', 'a', ['/', 'b', 'c']]]
ParseNode('~', None, [ParseNode('+', Token('+', <Origin a->+<-b / c (1-2)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<-+b / c (0-1)>, extra='a'), []), ParseNode('/', Token('/', <Origin a+b ->/<- c (4-5)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a+->b<- / c (2-3)>, extra='b'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a+b / ->c<- (6-7)>, extra='c'), [])])])])
'~ a' ['~', 'a']
ParseNode('~', Token('~', <Origin ->~<- a (0-1)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ~ ->a<- (2-3)>, extra='a'), [])])
'-1' ['~', ['-', '1']]
ParseNode('~', None, [ParseNode('-', Token('-', <Origin ->-<-1 (0-1)>), [ParseNode('ONE', Token('ONE', <Origin -->1<- (1-2)>, extra='1'), [])])])
'a | b' ['~', ['|', 'a', 'b']]
ParseNode('~', None, [ParseNode('|', Token('|', <Origin a ->|<- b (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- | b (0-1)>, extra='a'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a | ->b<- (4-5)>, extra='b'), [])])])
'a * b|c' ['~', ['*', 'a', ['|', 'b', 'c']]]
ParseNode('~', None, [ParseNode('*', Token('*', <Origin a ->*<- b|c (2-3)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin ->a<- * b|c (0-1)>, extra='a'), []), ParseNode('|', Token('|', <Origin a * b->|<-c (5-6)>), [ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a * ->b<-|c (4-5)>, extra='b'), []), ParseNode('PYTHON_EXPR', Token('PYTHON_EXPR', <Origin a * b|->c<- (6-7)>, extra='c'), [])])])])
a <+>
'a +' 2 3
expected a noun, but instead the expression ended
    a +
      ^
a + <(>
'a + (' 4 5
________________________________________________________ test_python_tokenize _________________________________________________________

    def test_python_tokenize():
        code = "a + (foo * -1)"
        tokens = list(python_tokenize(code))
        expected = [(tokenize.NAME, "a", Origin(code, 0, 1)),
                    (tokenize.OP, "+", Origin(code, 2, 3)),
                    (tokenize.OP, "(", Origin(code, 4, 5)),
                    (tokenize.NAME, "foo", Origin(code, 5, 8)),
                    (tokenize.OP, "*", Origin(code, 9, 10)),
                    (tokenize.OP, "-", Origin(code, 11, 12)),
                    (tokenize.NUMBER, "1", Origin(code, 12, 13)),
                    (tokenize.OP, ")", Origin(code, 13, 14))]
        assert tokens == expected

        code2 = "a + (b"
>       tokens2 = list(python_tokenize(code2))

patsy/tokens.py:74: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

code = 'a + (b'

    def python_tokenize(code):
        # Since formulas can only contain Python expressions, and Python
        # expressions cannot meaningfully contain newlines, we'll just remove all
        # the newlines up front to avoid any complications:
        code = code.replace("\n", " ").strip()
        it = tokenize.generate_tokens(StringIO(code).readline)
        try:
            for (pytype, string, (_, start), (_, end), code) in it:
                if pytype == tokenize.ENDMARKER:
                    break
                origin = Origin(code, start, end)
>               assert pytype != tokenize.NL
E               assert 65 != 65
E                +  where 65 = tokenize.NL

patsy/tokens.py:35: AssertionError

---------- coverage: platform linux, python 3.12.0-final-0 -----------
Name                             Stmts   Miss Branch BrPart  Cover   Missing
----------------------------------------------------------------------------
patsy/__init__.py                   58      0     12      2  97.1%   15->19, 57->exit
patsy/build.py                     314     16    168     10  93.8%   30, 683->685, 791->795, 869, 878->exit, 880-881, 896, 916-920, 926, 954-960
patsy/builtins.py                   20      0      0      0 100.0%
patsy/categorical.py               115     11     58      6  87.9%   52, 127, 194-195, 313-319, 338-339, 368
patsy/compat.py                     18      4      6      2  66.7%   29, 48-50
patsy/compat_ordereddict.py          9      1      0      0  88.9%   15
patsy/constraint.py                198      4     68      0  98.5%   15-16, 425-426
patsy/contrasts.py                 163      4     36      1  97.5%   78-81
patsy/desc.py                      205     10     56      6  93.1%   269, 319->325, 323-324, 326, 345, 354, 397, 402-407
patsy/design_info.py               322      2    172      2  99.2%   688, 1134
patsy/eval.py                      236      1     72      2  99.0%   106->109, 501
patsy/highlevel.py                 103     19     56      8  79.2%   28, 52-55, 157, 160, 192, 196-200, 215-216, 219-222
patsy/infix_parser.py              140      6     54      4  94.8%   45, 95, 138, 162, 185, 226
patsy/mgcv_cubic_splines.py        282      4    118      4  97.5%   17, 683-685, 713->716, 743->746
patsy/missing.py                    52      0     26      0 100.0%
patsy/origin.py                     36      3     10      0  93.5%   76, 85, 88
patsy/parse_formula.py              92      3     38      3  95.4%   54, 71, 142
patsy/redundancy.py                 91      2     28      0  98.3%   67, 105
patsy/splines.py                   100      4     48      2  94.6%   16, 243-245
patsy/state.py                      65      0     14      0 100.0%
patsy/test_build.py                 45      2      8      2  92.5%   25, 30
patsy/test_highlevel.py             28      9      4      0  59.4%   29, 59-62, 65-68
patsy/test_regressions.py            1      0      0      0 100.0%
patsy/test_splines_bs_data.py        4      0      0      0 100.0%
patsy/test_splines_crs_data.py       4      0      0      0 100.0%
patsy/test_state.py                 59     16     32      4  73.6%   42-63, 86-87, 102, 111
patsy/tokens.py                     77      4     34      2  94.6%   40, 44, 57-58
patsy/user_util.py                  69      0     22      0 100.0%
patsy/util.py                      224     49     82     11  69.9%   37, 47-55, 62-69, 161-164, 241-251, 299->298, 305->304, 449-450, 532, 615-623, 637-642, 649-654, 672, 678-682, 709
patsy/version.py                     1      0      0      0 100.0%
----------------------------------------------------------------------------
TOTAL                             3131    174   1222     71  92.9%
Coverage HTML written to dir /tmp/patsy/.tox/coverage/py312
Coverage XML written to file coverage.xml

======================================================= short test summary info =======================================================
FAILED patsy/desc.py::test_eval_formula_error_reporting - assert 65 != 65
FAILED patsy/parse_formula.py::test_parse_errors - assert 65 != 65
FAILED patsy/parse_formula.py::test_parse_extra_op - assert 65 != 65
FAILED patsy/tokens.py::test_python_tokenize - assert 65 != 65
============================================== 4 failed, 144 passed in 161.30s (0:02:41) ==============================================
py312: exit 1 (162.96 seconds) /tmp/patsy> pytest -vv --cov=patsy --cov-config=/tmp/patsy/.coveragerc --cov-report=term-missing --cov-report=xml --cov-report=html:/tmp/patsy/.tox/coverage/py312 pid=289499
.pkg: _exit> python /usr/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  py312: FAIL code 1 (198.75=setup[35.79]+cmd[162.96] seconds)
  evaluation failed :( (199.28 seconds)
mgorny commented 12 months ago

Ping.

matthewwardrop commented 12 months ago

I'll try to get this fixed shortly.

mgorny commented 12 months ago

Thanks a lot!

mgorny commented 12 months ago

Thank you! I can confirm that the new version works with py3.12.