openedx-unsupported / edx-analytics-pipeline

GNU Affero General Public License v3.0
91 stars 116 forks source link

Python 2/3 compatibility for finance tasks #747

Open pwnage101 opened 5 years ago

pwnage101 commented 5 years ago

This does not change any infrastructure to actually run anything under python 3, it only represents the results of my testing of the finance unit tests and acceptance tests under python 3 and modernizing the code to become both python 2 and 3 compatible.

Maybe half of the changes were caused by python-modernize (and manually audited), the other half are manual changes.


Testing requirements

The following four suite of tests are needed to pass before merging this PR:

suite status
Python 2 unit tests Passing.
Python 2 acceptance tests Failed: build #1795
Python 3 unit tests (financial tasks only) Passing.
Python 3 acceptence tests (test_financial_reports only) In Progress

Analytics Pipeline Pull Request

Make sure that the following steps are done before merging:

pwnage101 commented 5 years ago

Here's the current python 3 luigi execution summary:

20:40:00 ===== Luigi Execution Summary =====
20:40:00 
20:40:00 Scheduled 65 tasks of which:
20:40:00 * 3 present dependencies were encountered:
20:40:00     - 2 ExternalURL(url=s3+https://edx-analytics-test-data/credentials/acceptance-db.json,s3+https://edx-analytics-test-data/credentials/acceptance-vertica.json)
20:40:00     - 1 PaymentTask(...)
20:40:00 * 51 ran successfully:
20:40:00     - 1 FullOttoOrderTableTask(...)
20:40:00     - 1 FullShoppingcartOrderTableTask(...)
20:40:00     - 1 ImportAuthUserTask(...)
20:40:00     - 1 ImportCouponVoucherIndirectionState(...)
20:40:00     - 1 ImportCouponVoucherState(...)
20:40:00     ...
20:40:00 * 3 failed:
20:40:00     - 1 LoadInternalReportingFullShoppingcartOrdersToWarehouse(...)
20:40:00     - 1 OrderTableTask(...)
20:40:00     - 1 ReconcileFullOrdersAndTransactionsTask(...)
20:40:00 * 8 were left pending, among these:
20:40:00     * 6 had failed dependencies:
20:40:00         - 1 BuildEdServicesReportTask(...)
20:40:00         - 1 BuildFinancialReportsTask(...)
20:40:00         - 1 LoadInternalReportingEdServicesReportToWarehouse(...)
20:40:00         - 1 LoadInternalReportingFullOrderTransactionsToWarehouse(...)
20:40:00         - 1 ReconcileOrdersAndTransactionsTask(...)
20:40:00         ...
20:40:00     * 2 was not granted run permission by the scheduler:
20:40:00         - 1 LoadInternalReportingOrderTransactionsToWarehouse(...)
20:40:00         - 1 TransactionReportTask(...)