Logistic regression

[emercado4]

Pre-merge request checklist (to be completed by the one making the request):

• [x] I have performed a full review of this code myself.
  • For Python code in PySpark specific sections, all code should have been run in Jupyter notebooks.
  • For code in sections of the book containing both Python and R code, the page of the book should be constructed as described in the contributing guide and converted to a markdown file.
• [x] I have formatted the outputs of code blocks correctly (to match other outputs in the book and in line with the style guide [coming soon])
• [x] I have built the book as outlined in the contributing guide and confirmed that any additional/modified content is displaying as expected.

Details of this request:

New section on logistic regression with guidance on how this can be done in PySpark and sparklyr, issues to watch out for and how to access regression coefficients, confidence intervals etc.

Things to note about this request:

• The inferential analysis section PySpark example has quite a long-winded way of converting the summary table into a Pandas dataframe and adding in the confidence intervals. I expect there is a much quicker way of doing this and my lack of Python specific knowledge is preventing me from finding it so any suggestions would be appreciated :)
• In the pipeline section for sparklyr I have been unable to find a method that allows you to get a summary table which shows the feature names they correspond to clearly like you can for the non-pipeline version. Again, any suggestions for how we might be able to do this would be appreciated.

Requirements for review :

• [x] Code chunks review

  • [x] Check all code chunks run as expected
  • [x] Check that code is reasonably easy to follow
  • [x] Check that variables included in each step of analysis make sense to illustrate example properly (eg. sensible features are removed in the multicollinearity section etc.)
  • [x] Check that example outputs have been included where appropriate

• [x] Article content

  • [x] General checking for wording, writing flow, typos etc.
  • [x] Any incorrect information?
  • [x] Any unnecessary detail/information?
  • [x] Anything missing that would be important for understanding?
  • [x] Any references missing that would be helpful?
  • [x] Do all links work correctly?
  • [x] Confirm all tabs and expandable content works correctly

[ChrisSoderberg-ONS]

The logistic regression link hasn't been added to the table of contents file

[emercado4]

The logistic regression link hasn't been added to the table of contents file

Thanks for spotting - I've now added it to the Spark functions section but it might be moved to a new analysis section later.

[ChrisSoderberg-ONS]

I've checked the code again and confirmed that it works as expected, also added a converter script that turns this markdown file into a python notebook and r script so it can be run by others. I have verified that the book builds correctly as well

[emercado4]

Thanks so much for checking all this Chris! I've also now opened an issue (#104) for adding the cleaned rescue dataset to config.yaml so that the data reading process can be made consistent with the rest of the book.