The source mapper is very slow and a memory hog. As a result, two non-ideal artifacts have been added:
pyteal.ini which defaults to pyteal-source-mapper.enabled = False
Nightly tests in C.I. which carve out the slowest portion of the tests and complicate local testing
Solution
@jasonpaulos has an experimental approach which promises to speed things up to the point that these artifacts can be eliminated, or mitigated.
I propose:
nightly tests be eliminated once it is established that all tests can be run in a reasonable amount of time (however #630 should be taken into account before eliminating)
Default the sourcemapper to enabled = True in the pyteal.ini but don't get rid of it
Why keep the pyteal.ini ?
It may still be useful to allow dev's to turn the feature off completely.
We may desire to feature gate (cf. #601 ) via pyteal.ini. However, a more modern approach uses pyproject.toml instead.
Tasks
[ ] Improvements to NatalStackFrame and StackFrame
[ ] only keep the "best" frame, not a list of best frames (which in actuality always has length 1)
[ ] incorporate ideas from @jasonpaulos branch and this S.O. post
[ ] investigate saving memory by using slots (e.g.: nice example )
[ ] investigate replacing the debug-only member StackFrame.full_stack by StackFrame._debug_frame_origin and possibly add a method StackFrame._debug_full_stack() which can recreate this information
[ ] modify the unit tests that employ NatalStackFrame._keep_all_debugging so that they can work on the new lighter frame without keeping the entire stack. These are the following:
[ ] bring back python 3.10 source map tests into the "On Commit" workflow
[ ] either remove the nightly workflow, or repurpose it
[x] pyteal.ini (addressed in #687 and #688)
[x] decide whether to keep it and/or use pyproject.toml instead (DECISION: no file-based configs)
[x] if keeping, default source mapping to enabled
[x] or maybe support both: Adopt pytest's approach for dealing with both *.ini and pyproject.toml files. In particular, in the case that pyteal.ini isn't found, look also for pyproject.toml, and then for the section [tool.pyteal.ini] Such a section should look like:
Medium - defaulting to pyteal-source-mapper.enabled = True will make source mapping a lot more accessible, and getting rid of nightly tests will reduce developer friction.
Problem
The source mapper is very slow and a memory hog. As a result, two non-ideal artifacts have been added:
pyteal.ini
which defaults topyteal-source-mapper.enabled = False
Solution
@jasonpaulos has an experimental approach which promises to speed things up to the point that these artifacts can be eliminated, or mitigated.
I propose:
enabled = True
in thepyteal.ini
but don't get rid of itWhy keep the
pyteal.ini
?pyteal.ini
. However, a more modern approach usespyproject.toml
instead.Tasks
NatalStackFrame
andStackFrame
StackFrame.full_stack
byStackFrame._debug_frame_origin
and possibly add a methodStackFrame._debug_full_stack()
which can recreate this informationNatalStackFrame._keep_all_debugging
so that they can work on the new lighter frame without keeping the entire stack. These are the following:tests/unit/sourcemap_test.py::test_frames
tests/unit/sourcemap_monkey_unit_test.py::test_frame_info_is_right_before_core_last_drop_idx
pyteal.ini
(addressed in #687 and #688)pyproject.toml
instead (DECISION: no file-based configs)enabled
*.ini
andpyproject.toml
files. In particular, in the case thatpyteal.ini
isn't found, look also forpyproject.toml
, and then for the section[tool.pyteal.ini]
Such a section should look like:Dependencies
None
Urgency
Medium - defaulting to
pyteal-source-mapper.enabled = True
will make source mapping a lot more accessible, and getting rid of nightly tests will reduce developer friction.