frappe / erpnext

Free and Open Source Enterprise Resource Planning (ERP)
https://erpnext.com
GNU General Public License v3.0
21.99k stars 7.37k forks source link

Proposal for Snapshot Testing in ERPNext #43872

Open blaggacao opened 4 weeks ago

blaggacao commented 4 weeks ago

Introduction to Snapshot Testing

Snapshot testing is a type of regression testing that captures the current state of a system and compares it to a previously recorded state to identify changes. It is particularly useful for tracking changes in UI components, JSON data, or any other output format. This approach can simplify test writing, enhance test coverage, and automate manual processes.

Proposed Implementation

  1. Snapshot Capture and Storage:

    • Initiate snapshot mode in the UI during development.
    • Perform transactions and capture snapshots of records before and after changes.
    • Store snapshots in test_snapshots.toml files associated with the relevant doctype.
  2. Test Stub Creation:

    • Automatically generate a stub class, e.g., SnapshotTestMyDoctype, referencing snapshots as instance attributes.
  3. Transaction Recreation:

    • Developers specify transactions (e.g., create, validate) in tests using the captured snapshots.

Example Interaction

class SnapshotMyDoc(SnapshotTestCase):
    def __init__(self):
        self.snapshots = load([f for f in *.toml])

    def test_create_invoice_in_usd(self):
        so = self.snapshots[0]
        si = so.create_ainv()
        self.assertEqual(si, self.snapshots[1])

Challenges and Considerations

This proposal aims to streamline functional test creation in ERPNext by leveraging the benefits of snapshot testing.

Citations: [1] Snapshot Testing: Example and Its Benefits - ORIL https://oril.co/blog/snapshot-testing-example-and-its-benefits/ [2] Snapshot Testing in Action: Helpful or Trouble? - Snoopy Developer https://snoopydeveloper.com/snapshot-testing-in-action-helpful-or-trouble [3] Pros and cons of Jest snapshot testing | TSH.io - The Software House https://tsh.io/blog/pros-and-cons-of-jest-snapshot-tests/ [4] Snapshot Testing: Benefits and Drawbacks - SitePen https://www.sitepen.com/blog/snapshot-testing-benefits-and-drawbacks [5] When should I use Snapshot testing? - Stack Overflow https://stackoverflow.com/questions/43771602/when-should-i-use-snapshot-testing [6] Snapshot testing; what are the benefits? : r/reactjs - Reddit https://www.reddit.com/r/reactjs/comments/auyi54/snapshot_testing_what_are_the_benefits/

uc77 commented 4 weeks ago

Great initiative. We are with you on this.