feat!: Remove allow_certificate

[crice100]

Remove _allowcertificate since that column is being removed from the source db

DEPR-140 MICROBA-985

Analytics Pipeline Pull Request

Make sure that the following steps are done before merging:

• [ ] If you have a migration please contact data engineering team before merging.
• [ ] Before merging run full acceptance tests suite and provide URL for the acceptance tests run.
• [ ] A member of data engineering team has approved the pull request.

[codecov[bot]]

Codecov Report

Merging #863 (e7a1d06) into master (54cf01a) will decrease coverage by 0.00%. The diff coverage is 100.00%.

:exclamation: Current head e7a1d06 differs from pull request most recent head d7f21be. Consider uploading reports for the commit d7f21be to get more accurate results

@@            Coverage Diff             @@
##           master     #863      +/-   ##
==========================================
- Coverage   76.43%   76.43%   -0.01%     
==========================================
  Files         151      151              
  Lines       17170    17169       -1     
==========================================
- Hits        13124    13123       -1     
  Misses       4046     4046              

Impacted Files	Coverage Δ
...lytics/tasks/export/tests/test_data_obfuscation.py	97.58% <ø> (ø)
edx/analytics/tasks/tools/obfuscate_eval.py	0.00% <ø> (ø)
edx/analytics/tasks/export/data_obfuscation.py	89.28% <100.00%> (-0.03%)	:arrow_down:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 54cf01a...d7f21be. Read the comment docs.

[pwnage101]

There still seems to be a column count mismatch in one of the test fixtures, according to this error when I run acceptance tests:

15:21:09 Traceback (most recent call last):
15:21:09   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/analytics-tasks/edx/analytics/tasks/tests/acceptance/test_grades.py", line 30, in setUp
15:21:09     self.execute_sql_fixture_file('load_auth_userprofile.sql')
15:21:09   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/analytics-tasks/edx/analytics/tasks/tests/acceptance/__init__.py", line 331, in execute_sql_fixture_file
15:21:09     database.execute_sql_file(os.path.join(self.data_dir, 'input', sql_file_name))
15:21:09   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/analytics-tasks/edx/analytics/tasks/tests/acceptance/services/db.py", line 47, in execute_sql_file
15:21:09     for _ignored in cur.execute(sql_file.read(), multi=True):
15:21:09   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/build/venvs/analytics-tasks/lib/python2.7/site-packages/mysql/connector/cursor.py", line 437, in _execute_iter
15:21:09     iter(self._executed_list)):
15:21:09   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/build/venvs/analytics-tasks/lib/python2.7/site-packages/mysql/connector/connection.py", line 715, in cmd_query_iter
15:21:09     result = self._handle_result(self._socket.recv())
15:21:09   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/build/venvs/analytics-tasks/lib/python2.7/site-packages/mysql/connector/connection.py", line 601, in _handle_result
15:21:09     raise errors.get_exception(packet)
15:21:09 DataError: 1136 (21S01): Column count doesn't match value count at row 1

[pwnage101]

I tested test_grades.py, test_obfuscation.py, and test_module_engagement.py, and all succeeded:

• http://jenkins-ci.analytics.edx.org/view/ad-hoc/job/edx-analytics-pipeline-acceptance-test-manual/1908/
• http://jenkins-ci.analytics.edx.org/view/ad-hoc/job/edx-analytics-pipeline-acceptance-test-manual/1909/
• http://jenkins-ci.analytics.edx.org/view/ad-hoc/job/edx-analytics-pipeline-acceptance-test-manual/1910/

That seems good enough to me, and they were probably redundant since I was mostly just testing the fixture.

[pwnage101]

I'm having brianW to a look-over too, just to double check.

[pwnage101]

Started testing test_enterprise_user in http://jenkins-ci.analytics.edx.org/view/ad-hoc/job/edx-analytics-pipeline-acceptance-test-manual/1911/

[pwnage101]

I guess you can never stress the importance of testing enough...

19:59:07 ERROR: test_enterprise_user_table_generation (edx.analytics.tasks.tests.acceptance.test_enterprise_user.EnterpriseUserAcceptanceTest)
19:59:07 ----------------------------------------------------------------------
19:59:07 Traceback (most recent call last):
19:59:07   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/analytics-tasks/edx/analytics/tasks/tests/acceptance/test_enterprise_user.py", line 29, in setUp
19:59:07     self.prepare_database('lms', self.import_db)
19:59:07   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/analytics-tasks/edx/analytics/tasks/tests/acceptance/test_enterprise_user.py", line 44, in prepare_database
19:59:07     self.execute_sql_fixture_file(url_path_join(sql_fixture_base_url, filename), database=database)
19:59:07   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/analytics-tasks/edx/analytics/tasks/tests/acceptance/__init__.py", line 331, in execute_sql_fixture_file
19:59:07     database.execute_sql_file(os.path.join(self.data_dir, 'input', sql_file_name))
19:59:07   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/analytics-tasks/edx/analytics/tasks/tests/acceptance/services/db.py", line 47, in execute_sql_file
19:59:07     for _ignored in cur.execute(sql_file.read(), multi=True):
19:59:07   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/build/venvs/analytics-tasks/lib/python2.7/site-packages/mysql/connector/cursor.py", line 437, in _execute_iter
19:59:07     iter(self._executed_list)):
19:59:07   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/build/venvs/analytics-tasks/lib/python2.7/site-packages/mysql/connector/connection.py", line 715, in cmd_query_iter
19:59:07     result = self._handle_result(self._socket.recv())
19:59:07   File "/var/lib/jenkins/jobs/edx-analytics-pipeline-acceptance-test-manual/workspace/build/venvs/analytics-tasks/lib/python2.7/site-packages/mysql/connector/connection.py", line 601, in _handle_result
19:59:07     raise errors.get_exception(packet)
19:59:07 DataError: 1136 (21S01): Column count doesn't match value count at row 1

I'll take a look at a fix after my lunch

[brianhw]

Why not just run them all over the weekend? If you don't specify a specific test name, it will run on all tests....

[pwnage101]

oh good point, at first I was doing them one at a time thinking it would save time because there were only a few impacted tests, but it occurs to me now there are many. I'll fix this error then run them all over the weekend.

[pwnage101]

All acceptance tests passed: http://jenkins-ci.analytics.edx.org/view/ad-hoc/job/edx-analytics-pipeline-acceptance-test-manual/1912/