search

KingdomFirst / Bulldozer

A multi-component Windows app to convert data into Rock RMS from other church management systems.
https://www.rockrms.com/
Apache License 2.0
8 stars 9 forks source link

Bulldozer v2 #84

Closed guymaness2 closed 1 year ago

guymaness2 commented 1 year ago

Description

What does the change add or fix?

This PR constitutes a major rewrite of much of the Bulldozer application. The over arching philosophies behind this rewrite are:

Bulk Processing: Historically, Bulldozer has functioned by processing data in realtime as it iterates through every row of every csv file. This has worked reliably but at a performance cost. With this rewrite, Bulldozer uses a newer csv processing library to import entire csv files directly into Lists of c# classes, making the data much more efficient to work with. From there, new "import" classes are used as a sort of interface class to quickly translate csv data into objects ready to interact directly with Rock entities. This allows Bulldozer to take advantage of Rock's built in bulk processing framework, much like the core Bulk Import block does. This results in a significant performance improvement for migrations.

CSV File Restructure: Bulldozer has always worked from a strategic data structure based on csv columns. With this rewrite, we decided to align our csv structures with those RockRMS has already defined for the Bulk Import tool. The idea is to make Bulldozer a little more user friendly if the csv data is coming from a standardized Slingshot package from another system. This has resulted requiring more csv files than previously, but we feel that added element is worth the benefit of being able to process slingshot packages. That being said, as of this update, no Slingshot generated csv have been directly tested at this time, so we cannot yet guarantee they will work without any manipulation. That is the goal, though.

Work in Progress: As you may imagine, this is a large undertaking. It will take several iterations to touch everything we want to. Though most of the major data elements have been enhanced, there are still many that have not. They still function as they always have. We will continue to enhance the rest of the pieces we are targeting as we have time and resources to do so.

New data elements:

Optimized data elements:

Non-Optimized data elements:

Release Notes

What does the change add or fix in a succinct statement that will be read by clients?

[Add a bulleted list of release notes here]

Requested By

Who reported, requested, or paid for the change?

[Add the client here]

Screenshots

Does this update or add options to the block UI?

[Add a screenshot here]

Change Log

What files does it affect?

[Add a file list here]

Migrations/External Impacts

Is it a breaking change for other versions/clients?

[Add a no or yes with explanation here]

guymaness2 commented 1 year ago

Add info on which elements have been optimized and which are left to be done

nateh777 commented 1 year ago

@guymaness2 there are still unresolved conversations on this pull request. Can you please resolve them, by fixing what was brought up or comment on resolutions if they were resolved in a different manner. We will have one final review and then I will close this request and protect this branch from further modifications after that review.

nateh777 commented 1 year ago

This pull request was used to compare changes from BulldozerV2 to Hotfix-14. We are now going to close this pull request and leave BulldozerV2 as an open and protected branch, any further changes will be merged into BulldozerV2 using a pull request.