pcdshub / pytmc

Generate EPICS IOCs and records from TwinCAT projects - along with many TwinCAT project tools
https://pcdshub.github.io/pytmc/
Other
10 stars 11 forks source link

ENH: include type alias source pragmas in db generation #311

Closed ZLLentz closed 1 year ago

ZLLentz commented 1 year ago

Without this PR, type aliases don't "see" the pragmas from their type source. This means the database is generated with large numbers of missing records if we keep using the old names (DUT_MotionStage, DUT_PositionState) Database generation diff: https://github.com/pcdshub/lcls-plc-example-motion/pull/10/commits/6bbae6b30127325e0c5d012191152e551f4600b0

ZLLentz commented 1 year ago

Sorry, this definitely broke "something"- I should have slowed down before submitting. My new test passes but I forgot to check the rest of the long test suite.

ZLLentz commented 1 year ago

More detail: something I'm doing has managed to create unexpected bonus records. I'll dive deeper into it soon since this PR is definitely needed prior to deploying the ND motion thing.

``` _____________________________ test_allow_no_pragma _____________________________ def test_allow_no_pragma(): """ Test for the existence of a variable included in the records, despite it lacking a proper set of pragmas. """ tmc_file_name = TMC_ROOT / ("xtes_sxr_plc.tmc") tmc = parser.parse(tmc_file_name) records, exceptions = db_process( tmc, dbd_file=None, allow_errors=True, show_error_context=True, allow_no_pragma=False, ) all_records, exceptions = db_process( tmc, dbd_file=None, allow_errors=True, show_error_context=True, allow_no_pragma=True, ) good_records = 129 total_records = 1002 assert good_records == len(records) assert good_records == len(list(x.valid for x in records if x.valid)) > assert total_records == len(all_records) E AssertionError: assert 1002 == 1005 E + where 1005 = len([RecordPackage(tcname='.TCPADS_MAXUDP_BUFFSIZE'), RecordPackage(tcname='Constants.CompilerVersion.uiMajor'), RecordPackage(tcname='Constants.CompilerVersion.uiMinor'), RecordPackage(tcname='Constants.CompilerVersion.uiPatch'), RecordPackage(tcname='Constants.CompilerVersion.uiServicePack'), RecordPackage(tcname='Constants.CompilerVersionNumeric'), ...]) ```
klauer commented 1 year ago

What are the new records it finds?

ZLLentz commented 1 year ago

What actually happens here is that the library finds 3 new record candidates in the test case. These are all of type AMSNETID which, has BYTE as its base type. Since we now check base types, and since BYTE is something we could make a record of, it is now found. image

klauer commented 1 year ago

Seems like a bug - though I'm curious what our IOC databases will look like after this PR. Any change?

ZLLentz commented 1 year ago

Note that this test makes 3 new records only when allow_no_pragma=True, not for the normal case.

Personally, my database didn't have any weird bonus records other than reinstating the PVs for DUT_MotionStage, etc. I didn't accumulate any AMSNETID records because none of mine are pragma'd (I think?)

Two tests I want to run:

  1. Run pytmc tagged vs this PR on a few real projects
  2. Include an AMSNETID variable with a pragma to see what happens before and after this diff
ZLLentz commented 1 year ago

I did test 1 for all of the production beamline plcs that I could quickly remember, provided pytmc master branch ran without errors. This excludes lfe and kfe vac which do not build iocs right now- I will make an issue and investigate separately.

For all the others, this did not change their record generation at all. (Note: no production plc is running the unreleased motion lib that turns DUT_MotionStage into an alias.)

You can snoop on my results at /cds/home/z/zlentz/temp/pytmc_testing/pytmc_testing.diff and in that folder for my quick/messy metholodoly

ZLLentz commented 1 year ago

Test 2 had more mixed results.

I created an AMSNETID instance in lcls-plc-example-motion and added a pragma (https://github.com/pcdshub/lcls-plc-example-motion/commit/7638617a5e3f0366917ce10839937689bd281892)

Building with pytmc master did not add any PVs pointing to this, which makes sense given what we see in the test suite. Building with this PR finds the symbol and tries to make a record but gets stuck:

``` ERROR:pytmc.bin.template:Failed to create EPICS records Traceback (most recent call last): File "/cds/home/z/zlentz/pydev/pytmc/bin/template.py", line 374, in get_plc_record_packages packages, exceptions = db_process( File "/cds/home/z/zlentz/pydev/pytmc/bin/db.py", line 154, in process record_names = [ File "/cds/home/z/zlentz/pydev/pytmc/bin/db.py", line 158, in for single_record in record_package.records File "/cds/home/z/zlentz/pydev/pytmc/record.py", line 619, in records records = [self.generate_input_record()] File "/cds/home/z/zlentz/pydev/pytmc/record.py", line 955, in generate_input_record record = super().generate_input_record() File "/cds/home/z/zlentz/pydev/pytmc/record.py", line 511, in generate_input_record record.fields["DTYP"] = self.dtyp File "/cds/home/z/zlentz/pydev/pytmc/record.py", line 952, in dtyp return data_types[self.chain.data_type.name] KeyError: 'AMSNETID' Traceback (most recent call last): File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.7.2/bin/pytmc", line 10, in sys.exit(main()) File "/cds/home/z/zlentz/pydev/pytmc/bin/pytmc.py", line 106, in main func(**kwargs) File "/cds/home/z/zlentz/pydev/pytmc/bin/template.py", line 783, in main raise stashed_exception File "/cds/home/z/zlentz/pydev/pytmc/bin/template.py", line 776, in main rendered = render_template(template_text, template_args) File "/cds/home/z/zlentz/pydev/pytmc/bin/template.py", line 660, in render_template return env.get_template("template").render(context) File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.7.2/lib/python3.9/site-packages/jinja2/environment.py", line 1301, in render self.environment.handle_exception() File "/cds/group/pcds/pyps/conda/py39/envs/pcds-5.7.2/lib/python3.9/site-packages/jinja2/environment.py", line 936, in handle_exception raise rewrite_traceback_stack(source=source) File "