Updates for Tax-Calculator 3.0

[hdoupe]

This updates the cs-config directory and the taxbrain package to work with ParamTools styled inputs:

• Moves inputs-related utils functions to cs_config/inputs.py. This will make it easier to share the utility functions needed to use Tax-Calculator on Compute Studio with other projects (or possibily break it off into its own light-weight package).
• Adds a couple helper functions in taxbrain/utils.py that determine whether a reform is likely to be in the Tax-Calculator or ParamTools format.

I need to write tests for the new functions in utils.py but for the most part, this should be ready to go.

[hdoupe]

@andersonfrailey I got this test failure. I think it might be related to updated results from taxcalc:

tb_static = <taxbrain.taxbrain.TaxBrain object at 0x7f27d72e8410>

    def test_weighted_totals(tb_static):
        table = tb_static.weighted_totals("combined")
        assert isinstance(table, pd.DataFrame)
        # table.to_csv("expected_weighted_table.csv")
        cur_path = os.path.dirname(os.path.abspath(__file__))
        expected_table = pd.read_csv(os.path.join(cur_path,
                                                  "expected_weighted_table.csv"),
                                     index_col=0)
        # convert columns to integers to avoid a meaningless error
        expected_table.columns = [int(col) for col in expected_table.columns]
        diffs = False
        for col in table.columns:
            if not np.allclose(table[col], expected_table[col]):
                diffs = True
        if diffs:
            new_file_name = "actual_weighted_table.csv"
            table.to_csv(os.path.join(cur_path, new_file_name))
            msg = ("Weighted table results differ from expected. New results are"
                   " in actual_weighted_table.csv. If new results are ok, copy"
                   " actual_weighted_table.csv to expected_weighted_table.csv"
                   " and rerun test.")
>           raise ValueError(msg)
E           ValueError: Weighted table results differ from expected. New results are in actual_weighted_table.csv. If new results are ok, copy actual_weighted_table.csv to expected_weighted_table.csv and rerun test.

taxbrain/tests/test_brain.py:61: ValueError

[hdoupe]

@andersonfrailey I'm also getting a test failure in the policy_table function, too. This is from the updated defaults file structure:

        pol = tc.Policy()  # policy object used for getting original value
        # loop through all of the policy parameters in a given reform
        for param, meta in params.items():
            # find all the years the parameter is updated
            years = set(meta.keys())
            reform_years = reform_years.union(years)
            for yr in years:
                # find default information
                pol.set_year(yr)
                pol_meta = pol.metadata()[param]
>               name = pol_meta["long_name"]
E               KeyError: 'long_name'

taxbrain/report_utils.py:133: KeyError

I can fix the key error, but I'm not sure how to update vi_name in the following code snippet:

https://github.com/PSLmodels/Tax-Brain/blob/517469df91bcd8f36ea864dea2230270f0e8ade6/taxbrain/report_utils.py#L133-L152

[andersonfrailey]

Thanks for working on this, @hdoupe. The first error you can just copy/paste over the new table into the expected table. This error is expected with the new version of the PUF/CPS.

As for the second, do you know which parameter is causing the error?

[hdoupe]

@andersonfrailey All tests are passing except for the reports test. I think this test is failing since TC 3.0 is now using ParamTools and the parameters have a little bit different format. I gave you write access to my branch, and welcome any help to get that to pass. I'm jammed up this week but will try to circle back soon.

[andersonfrailey]

Thanks, @hdoupe! I'll take a crack at it later this week

[Peter-Metz]

Hey guys, I tried poking around this issue but cant get the report to run locally with Tax-Calc 2.9. I've run in to the following pandoc error and installing latex and pdflatex didn't help. Have you seen this before?

================================================== test session starts ===================================================
platform darwin -- Python 3.7.8, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/petermetz/hank/Tax-Brain
plugins: shutil-1.7.0
collected 1 item                                                                                                         

test_report.py F                                                                                                   [100%]

======================================================== FAILURES ========================================================
______________________________________________________ test_report _______________________________________________________

tb_static = <taxbrain.taxbrain.TaxBrain object at 0x7fa9d6f7a8d0>

    def test_report(tb_static):
        """
        Ensure that all report files are created
        """
        outdir = "testreform"
        name = "Test Report"
        report(
>           tb_static, name=name, outdir=outdir
        )

test_report.py:13: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../report.py:210: in report
    md_to_pdf(report_md, str(pdf_path))
../report_utils.py:65: in md_to_pdf
    '-V', 'geometry:margin=1in'
../../../../anaconda3/envs/taxbrain-dev/lib/python3.7/site-packages/pypandoc/__init__.py:101: in convert_text
    outputfile=outputfile, filters=filters)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

source = b'% Test Report\n% \n% October 5, 2020\n\n## Table of Contents\n\n### [Introduction](#Introduction)\n* [Analysis Summa...ic models:\n\n\n* Tax-Brain release 0.0.0\n\n* Tax-Calculator release 2.9.0\n\n* Behavioral-Responses release 0.10.0\n'
format = 'markdown', input_type = 'string', to = 'latex', extra_args = ['-V', 'geometry:margin=1in']
outputfile = 'testreform/Test-Report.pdf', filters = None

    def _convert_input(source, format, input_type, to, extra_args=(), outputfile=None,
                       filters=None):
        _ensure_pandoc_path()

        format, to = _validate_formats(format, to, outputfile)

        string_input = input_type == 'string'
        input_file = [source] if not string_input else []
        args = [__pandoc_path, '--from=' + format]

        args.append('--to=' + to)

        args += input_file

        if outputfile:
            args.append("--output=" + outputfile)

        args.extend(extra_args)

        # adds the proper filter syntax for each item in the filters list
        if filters is not None:
            if isinstance(filters, string_types):
                filters = filters.split()
            f = ['--filter=' + x for x in filters]
            args.extend(f)

        # To get access to pandoc-citeproc when we use a included copy of pandoc,
        # we need to add the pypandoc/files dir to the PATH
        new_env = os.environ.copy()
        files_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "files")
        new_env["PATH"] = new_env.get("PATH", "") + os.pathsep + files_path

        p = subprocess.Popen(
            args,
            stdin=subprocess.PIPE if string_input else None,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            env=new_env)

        # something else than 'None' indicates that the process already terminated
        if not (p.returncode is None):
            raise RuntimeError(
                'Pandoc died with exitcode "%s" before receiving input: %s' % (p.returncode,
                                                                               p.stderr.read())
            )

        try:
            source = cast_bytes(source, encoding='utf-8')
        except (UnicodeDecodeError, UnicodeEncodeError):
            # assume that it is already a utf-8 encoded string
            pass
        try:
            stdout, stderr = p.communicate(source if string_input else None)
        except OSError:
            # this is happening only on Py2.6 when pandoc dies before reading all
            # the input. We treat that the same as when we exit with an error...
            raise RuntimeError('Pandoc died with exitcode "%s" during conversion.' % (p.returncode))

        try:
            stdout = stdout.decode('utf-8')
        except UnicodeDecodeError:
            # this shouldn't happen: pandoc more or less garantees that the output is utf-8!
            raise RuntimeError('Pandoc output was not utf-8.')

        # check that pandoc returned successfully
        if p.returncode != 0:
            raise RuntimeError(
>               'Pandoc died with exitcode "%s" during conversion: %s' % (p.returncode, stderr)
            )
E           RuntimeError: Pandoc died with exitcode "47" during conversion: b'pdflatex not found. Please select a different --pdf-engine or install pdflatex\n'

../../../../anaconda3/envs/taxbrain-dev/lib/python3.7/site-packages/pypandoc/__init__.py:321: RuntimeError
================================================ short test summary info =================================================
FAILED test_report.py::test_report - RuntimeError: Pandoc died with exitcode "47" during conversion: b'pdflatex not fou...
=================================================== 1 failed in 35.12s ===================================================

[andersonfrailey]

@Peter-Metz, I have not. But I'm working on general fixes for the report feature right now so I'll let you know if I run into it.

[hdoupe]

@andersonfrailey this is ready to go!