search

banillie / analysis_engine

Place for all code used to compile the quarterly PfM Report, manage GMPP data, as well as other useful data searching/analysis functions.
MIT License
2 stars 2 forks source link

Mentoring. Things to cover. #21

Closed banillie closed 3 years ago

banillie commented 3 years ago

@yulqen thought I'd note down some of the things it would be great to cover in our mentoring, other than the other issues we've raised.

GitHub git push for other repos I've recently set up requires me to do user name and password. Getting these repos to use encryption key. DONE

Some tidying of analysis_engine repo and what's included in git. DONE

I've created a couple of new repos for other reporting processes. Both use analysis_engine code. I've pip installed ae into the virtualenvs for these repos, but probably a sub-optional way of working. discussed and going to integrate new repos into ae DONE

Airtable. Show me how it works. Had a look at it and was struggling! STOPPED

Testing. I'm updating the unit tests to use a pickle file (for now). However, I've got a number of live tests, currently in the scatter_chart file, which are useful, because they do live testing. You referred to these as another sort of test. Can't remember the name you used. Lets discuss how I can formalize these tests.

Cheers.

yulqen commented 3 years ago

On using your ssh keys to authenticate on Github:

Make sure you still have a key still registered on your account (Go to the security settings on Github) to check.

Check your private key is still on your machine: ls ~/.ssh/ - there is probably two key files there named something like id_rsa.pub and id_rsa. The .pub key is the one you pasted into the Github security page.

Make sure your git configuration in the projects you are talking about are using the correct URL format so that git push uses the SSH protocol rather than https. This is the most likely explanation:

Do cat path_to_project/.git/config. Check the url param inside the [remote "origin"] block uses a URL of the format git@github.com:banillie/project_name.git and not a URL that starts with https. If this is the case, you need to change the URL to the correct format - either amend the .git/config file directly or do git remote set-url origin git@github.com:banillie/project_name.git.

yulqen commented 3 years ago

It's fine to pip install analysis_engine as a tool to help you with other projects. You only need to consider a different approach and start to declare ae as a dependency if you're going to be putting these new projects up on PyPI.

Also - happy to show you Airtable, although I've not looked at it since I mentioned it!

Come to our session on Friday with a list of stuff you need me to look at - look forward to it.

banillie commented 3 years ago

Thanks. using ssh key for other repos worked perfectly.

I plan to put these new programs into packages. There is a wider discussion with you here actually, because these packages will be doing very similar things to ae, but are using different data sets. Would be good to discuss what you think. At the moment I'm hacking my way through and learning via doing, and hopefully some ideas for a refactor and how to better integrate will come soon.

When I do package them up would the ae dependency sit in the setup.py doc under install requirements. Plus I'd probably create a requirements.txt file.

I was wondering if there is a way to use my local ae project directly in the other projects virtualenv. So if I make a change to the ae code its immediately picked up in other projects virtualenv.

Will have a think about things to discuss. Cheers man. looking forward to it also.

yulqen commented 3 years ago

We will have to discuss these things on Friday.

setup.py is for packaging and pip. requirements.txt is for virtualenvs, etc. We can discuss when I know more about what you want to do.

The idea to use your analysis_engine project inside another project doesn't sound like a good idea! I'm sure I can solve this if I can understand what you actually need.

banillie commented 3 years ago

@yulqen have updated the initial issue, following our latest mentoring session. Do you get notifications when the previous comments are edited?

yulqen commented 3 years ago

Yes!

On 28-04-2021, Will Grant wrote: @.*** have updated the initial issue, following our latest mentoring session. Do you get notifications when the previous comments are edited?

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/banillie/analysis_engine/issues/21#issuecomment-828267508

--

Matthew Lemon Email: @.***

banillie commented 3 years ago

Have updated the initial post here. The only that remains outstanding is discussing different testings options/models.

yulqen commented 3 years ago

I think I was referring to integration testing, or end-to-end testing. That usually means something more than testing the small pieces of code/functions that constitute "unit testing". The strategy here would be to use pytest.mark to "mark" tests in your test suite that should be run separately from your unit tests (see https://docs.pytest.org/en/6.2.x/skipping.html). You can essentially flag all tests that produce graphical output with a tag or keyword (such as "graphs"), and then when you run your tests you can do pytest -k "not graphs", which will tell pytest to skip all tests. This will encourage you to run the unit tests more often and they will run quickly and without all the graphic windows popping up. Have a look at examples of how I use pytest.mark in bcompiler.engine

banillie commented 3 years ago

thanks for this. looking into it.

banillie commented 3 years ago

@yulqen thanks for the above. I've changed the tests so that no chart output is displayed as part of test. Therefore, all 61 tests, run in around 11 seconds and don't require the manual closing of charts.

I've also started to use the pytest.mark to mark tests that shouldn't be run are failing.