REF/ENH: rework Genesis 4 support completely (lark grammar-based parser + pydantic)

ken-lauer commented 6 months ago

Description

Add lark grammar for Genesis 4 main input file format (namelists)
Add lark grammar for Genesis 4 lattice input file format
Add documentation parsing utilities for Genesis 4 manuals
Add automatically-generated Pydantic models for main input file and lattice input file objects (namelists and beamline elements)
- Model instances are generated when parsing input files
- Model classes may also be used standalone
- Model instances can be easily serialized back to the original file formats
Supports references where and when Genesis does

TODO / plan details

[x] Move to v4 subdirectory
[x] Write round-trip tests (file -> dataclass -> file -> dataclass and compare)
[x] Change classes with _ in them to CamelCase; track original &name_list name as a class attribute
[x] Update manual based on fixes
[x] Propose manual fixes back to Genesis 4
[x] Read genesis code and understand where references are allowed
[x] Implement references (field/lambda may be the only that remains to fix in original doc)
[x] Support old JSON format for (de)serialization
[x] Rename Genesis4Python to Genesis4 and back-compat as much as possible/feasible
[x] Support archiving of input/output/run data
[x] Address discrepancy between 'path' and 'workdir' in Genesis4 for new method
[x] When lattice is not specified and just main input is specified, determine lattice filename based on parsed main input
[x] At least some Jupyter-specific rendering of objects
[x] Add documentation for end-users ("how to construct lattice from these elements...")
[x] Incorporate parsing/writing into documentation
[x] Add more example jupyter notebooks
[x] Add initial_particles setter with defaults to set_initial_particles(update_slen=True) (etc.) to MainInput
[x] Add InitialParticles tests to the test suite
[x] Verify reasonable test coverage

Chris TODO

[x] Consult with @ChristopherMayes to determine end-user focused ease of use changes
[x] @ChristopherMayes needs to take a look at the renames used for attribute names in the main input namelists and the lattice beamline components

Future (not in the scope of this PR)

Propose new site for genesis 4 docs
Wavefront object in pmd_beamphysics for loaded fields
Propose JSON as parameter information source upstream
- Usage of namelists and lattice elements will be in JSON
- Manual will be generated based on that
- We will use that packaged file generating our models

ChristopherMayes commented 6 months ago

This is for Genesis4 only, so the lark and grammar files should go in genesis/version4

ChristopherMayes commented 6 months ago

How were the data classes made? I would be better if the comments were part of the docstring.

ken-lauer commented 6 months ago

How were the data classes made?

I didn't get to the JSON version of the upstream documentation as I was waiting on the other PR. I think hand-crafted will always win when compared to auto-generated - especially in the case where these things don't change very often - but that's another thing we can discuss and weigh up.

I would be better if the comments were part of the docstring.

For sphinx, #: syntax will include such comments as part of the generated documentation for each attribute.

It's partly a stylistic choice, partly a functional one. Personally, I think keeping the description close to the declaration rather than away in a docstring that you have to search through (and may more easily get out of sync) is preferable.

ChristopherMayes commented 6 months ago

I'd like to have the field descriptions accessible without having to go to the source file, so they should at least be in the docstring. Similar to the main input structures, I think we can automate all of this from JSON info, I'm not sure how much needs to be handcrafted.

ken-lauer commented 6 months ago

Turns out there are a bunch of parameters missing in the documentation for the main input file format (specifically "track"; others will have to be audited similarly). I'll start to fix the upstream documentation first to get this fully compliant - though I'm not sure I'll be able to figure out the parameter descriptions on my own.

Edit: Here's a few more, found by grepping and checking versus our dataclasses in this PR:

Altersetup parameter disable missing
Setup parameter outputdir missing
Setup parameter exclude_field_dump missing
Setup parameter write_meta_file missing
Setup parameter write_semaphore_file missing
Setup parameter write_semaphore_file_done missing
Setup parameter write_semaphore_file_started missing
Track parameter s0 missing
Track parameter slen missing
Track parameter field_dump_at_undexit missing
Track parameter bunchharm missing
Track parameter dbg_report_lattice missing
Track parameter dbg_suppress_outfile missing

Edit 2:

Second audit of sorts - missing parameters listed in namelist "usage()" functions:

Missing field for class Profile_file: autoassign
Missing field for class Sequence_power: seed
Missing field for class Sequence_power: normal
Missing field for class Importdistribution: settimewindow
Missing field for class Importdistribution: eval_start
Missing field for class Importdistribution: eval_end
Missing field for class Importdistribution: align
Missing field for class Importdistribution: align_start
Missing field for class Importdistribution: align_end
Missing field for class Setup: outputdir
Missing field for class Setup: exclude_field_dump
Missing field for class Setup: write_meta_file
Missing field for class Setup: write_semaphore_file
Missing field for class Setup: write_semaphore_file_done
Missing field for class Setup: write_semaphore_file_started
Missing field for class Setup: semaphore_file_name
Missing field for class Altersetup: string
Missing field for class Altersetup: disable
Missing field for class Track: s0
Missing field for class Track: slen
Missing field for class Track: field_dump_at_undexit
Missing field for class Track: bunchharm
Missing field for class Efield: dump

Also: alter_setup is named incorrectly (patching manual here; will propose upstream as well)

Need to double-check those against the parsers...

ChristopherMayes commented 6 months ago

That's okay. Put in a placeholder to be filled out later.

ChristopherMayes commented 1 month ago

I see a Numpy v2 error running the Example1 notebook:

AttributeError: `np.string_` was removed in the NumPy 2.0 release. Use `np.bytes_` instead.

ken-lauer commented 1 month ago

I see a Numpy v2 error running the Example1 notebook:
AttributeError: `np.string_` was removed in the NumPy 2.0 release. Use `np.bytes_` instead.

There isn't a string_ in this codebase; it may be coming from another library (did you upgrade openpmd-beamphysics?).

ChristopherMayes commented 1 month ago

@ken-lauer sorry, I was using the wrong branch.

ChristopherMayes commented 1 month ago

This has 96 hardcoded, which is too high for most people. Also it's inconsistent with text above:

ken-lauer commented 1 month ago

This has 96 hardcoded, which is too high for most people

There's a warning above:

Warning this example requires a considerable amount of memory (over 100GB) and processor cores in order to be run in a reasonable amount of time. Your laptop may not be sufficient.

ChristopherMayes commented 1 month ago

What about the text about 12 cores in there?

ken-lauer commented 1 month ago

What about the text about 12 cores in there?

Where?

ChristopherMayes commented 1 month ago

slaclab / lume-genesis