Update manual

[bjbvoesenek]

Hey Ivo, I wrote a manual, see the readme.md file. I also looked up versions of python and it's libraries we have used.

---

Copied from PR #16, which was merged before completion:

Work In Progress (WIP): Documentation for v2.

We can work together in this branch to finish the documentation, as long as we communicate with each other when to push or pull the code.

We can easily reformat this manual in any way you see fit for your project. I do highly recommend keeping it in version control, in other words, keep using Markdown. We can always generate PDFs from this Markdown. You can see the Markdown rendering of this branch here.

Left to do:

• [x] Add versions of requirements? See the TODO in the text.
• [x] What Python version do we need?
• [x] Standard Python libraries can be removed from the list.
• [x] Perhaps the project needs a requirements.txt for pip to work with?
• [x] Explain each output file? See the TODO in the text.
• [x] Should we include screenshots etc? If so, we'll need a separate directory to dump those pictures in, e.g., a docs folder.
• [x] The Markdown can be rendered into PDF; I can add the instructions on how to do so in the code of the Markdown without being visible in the text.

[ifokkema]

We now have two branches with an update to the manual, each with its own PR. This is for 2.1, but linked to the 2.0 release. However, our 2.0 manual branch is v2/docs branch (PR #16). This will cause problems, as we now have two different manuals from the same point of origin (release/2.0). Normally, we'd finish the 2.0 manual first, merge that, and only then start working on the 2.1 manual. Now we'll need to manually merge these two versions. I assume you don't have any changes for the 2.0 manual in v2/docs branch? In that case, I will merge that PR, move all remaining requirements to this PR, and then try to untangle the code. I think the best way forward is to forget about 2.1 completely and include all the work done in 2.0. It'll save me part of the untangling since I no longer need to split 2.0 and 2.1, but it will further delay releasing 2.0.

Note, that the 2.0 release is anyway blocked by changes required for #8, but I can start trying to merge the manuals already. Please let me know when you are done with changes, so I can start working on it.

[bjbvoesenek]

I only added the versions to your readme in v2/docs, for the rest I didn't have any changes

[bjbvoesenek]

I can also add thy python manual with screenshots to v2/docs

[ifokkema]

Full list of versions copied from the original PR for the 2.0 manual:

Hereby the versions of python and packages used:

python 3.6.13 matplotlib 3.3.4 natsort 8.2.0 numpy 1.19.5 pandas 0.23.0 pip 21.2.2 xlrd 1.2.0

For the following packages I can't find a version, but as they are python build-in packages there maybe is no need to mention versions of these packages. I saw that others also don't list these: argparse math os sys statistics

[ifokkema]

I can also add thy python manual with screenshots to v2/docs

No, please don't copy anything - using Git, you should never have to copy anything. Git has all the power you need to take the actual changes instead of redoing things more than once. Note that every line has a history, an author, a timestamp... Doing things more than once breaks all of these things.

[bjbvoesenek]

About Maurice's manual, I am not sure how this one is licensed, that's why I did not include his manual or content of it. I can ask today.

I'll think about the requirements today. One the one hand, I would suggest to stimulate users to use the web-interface. You put a lot of work in it and it looks amazing, plus you can make less mistakes using the web-interface compared to command-line. On the other hand, it would be useful if people really want to work with the command-line.

I'll have a good look at your manual to make changes according to these points in the future ;)

[bjbvoesenek]

OK, the diff looks clean now, so we have a nice and clean version of the manual to work with.

Things left to do:

• [ ] Perhaps the project needs a requirements.txt for pip to work with? I'll leave this decision up to you completely.
• [ ] Please check my comments for each file. I have added a description for each file according to how I understand it, but I may have gotten things wrong.
• [ ] I believe all the links in the manual work but feel free to verify or add links when needed. Links help move the user through the documentation without the need to duplicate text. This keeps the manual concise and helps getting a better overview (IMHO).
• [ ] Maurice's manual is mentioned twice, but there's no way for external users to get it. If the relevant steps from the manual can not be included here, then we should put that manual online (perhaps on the Transpose page?) and refer to it. Otherwise, the relevant steps should be copied, and then we don't need to refer to it.

Some notes:

• Since Git considers every line as a "unit", a single character change will highlight the whole line as changed. Therefore, it's important to keep every new sentence on a new line in the source of the manual, so that when one character changes, only one sentence is marked as changed.
• Also, it's common practice to keep lines to a certain maximum number of characters. This is especially true for documentation, which is often written with a rendered version displayed next to the source code, and there is limited space to show the code. A very common maximum line length is 80 characters.

I edited the manual to follow both recommendations.

• Also, blinded BioRad input in readme.md

[ifokkema]

About Maurice's manual, I am not sure how this one is licensed, that's why I did not include his manual or content of it. I can ask today.

That would be great! Otherwise, we could simply rewrite the relevant steps ourselves. As long as it's not a copy but our own work, there will be no license issues. And otherwise; Maurice's supervisor can decide on a license, even if it hasn't been licensed before. Maurice himself isn't the copyright holder; the LUMC is.

I'll think about the requirements today. One the one hand, I would suggest to stimulate users to use the web-interface. You put a lot of work in it and it looks amazing, plus you can make less mistakes using the web-interface compared to command-line. On the other hand, it would be useful if people really want to work with the command-line.

Sure, and we can focus on letting people use our installation of the interface, but in case others want to run the web interface themselves (e.g., because of privacy issues), they'll need to install the Python script and the web interface themselves, at their institute. It shouldn't be too much work to add the requirements.txt, but if you don't feel like creating, testing, supporting, and maintaining it, just leave it out :wink:

I'll have a good look at your manual to make changes according to these points in the future ;)

Great, thanks!

[bjbvoesenek]

I added Maurice his qPCR manual, so I think we can remove the TODO's from the readme

[bjbvoesenek]

Only the requirements are left to do, right?

I will make an issue with ideas for future releases

[ifokkema]

I added Maurice his qPCR manual, so I think we can remove the TODO's from the readme

Thanks! I reviewed Maurice's manual again and determined only pages 12 - 14 are useful. The rest describes steps not relevant to this project. Or am I missing something? Because otherwise, we could include pages 12 - 14 instead of the whole file, or we could simply include the text and pictures. It seems a bit strange to refer to an external file for a few pages and then return to our own readme to continue the process. Also, it seems that part of the instructions in the manual should not be followed (like adding more tabs to the Excel file). But please check my reasoning - I could be misunderstanding the PDF. Please also check what I changed in the manual.

[ifokkema]

Only the requirements are left to do, right?

I have added the requirements.txt and referred to it in the manual: https://github.com/bjbvoesenek/qPCR-analysis/pull/18/commits/bc9f356cde92afea7957c661cac076834fc4a6af

I will make an issue with ideas for future releases

Excellent!

I still have open to let the Python script add the version to the output, which then also should be included in the manual and the web interface. That's the last thing, indeed. Then also #8 can be closed.

[ifokkema]

I still have open to let the Python script add the version to the output, which then also should be included in the manual and the web interface. That's the last thing, indeed. Then also #8 can be closed.

Do I understand correctly from #19 that you don't want the version of the script included in anything yet? (manual, web interface) But instead, move that to a later release? I recommend still including it since it's very little work, but it's important for reproducibility. I can even do it myself if you want.

[bjbvoesenek]

I added Maurice his qPCR manual, so I think we can remove the TODO's from the readme

Thanks! I reviewed Maurice's manual again and determined only pages 12 - 14 are useful. The rest describes steps not relevant to this project. Or am I missing something? Because otherwise, we could include pages 12 - 14 instead of the whole file, or we could simply include the text and pictures. It seems a bit strange to refer to an external file for a few pages and then return to our own readme to continue the process. Also, it seems that part of the instructions in the manual should not be followed (like adding more tabs to the Excel file). But please check my reasoning - I could be misunderstanding the PDF. Please also check what I changed in the manual.

Cool, will check your link tomorrow! One the one hand, we can indeed only add pages 12-14 to our git, if this prevents confusion. On the other hand, the calculations done by the script are explained in this manual (except for the N0, we don't use that). Might be nice for people that want to know how the relative gene expression is calculated, but have no idea how to read the code. Either way, we should explain what kind of information is in the qPCR manual, by mentioning it somewhere in the readme (.. relative gen expression is calculated as described in the qPCR manual).

I agree with shortening the manual, do you think that we should only include page 12-14 (will check tomorrow what is in page 12-14😂) or also some background about the script's calculations?

[bjbvoesenek]

I still have open to let the Python script add the version to the output, which then also should be included in the manual and the web interface. That's the last thing, indeed. Then also #8 can be closed.

Do I understand correctly from #19 that you don't want the version of the script included in anything yet? (manual, web interface) But instead, move that to a later release? I recommend still including it since it's very little work, but it's important for reproducibility. I can even do it myself if you want.

Feel free to do it if you know how! Otherwise I can have a look.

edit: I see you already implemented it, great!

[ifokkema]

Updated the online platform, so you can show it live on Monday! https://humgen.nl/scripts/qPCR-analysis/

I'll also send you a new PDF with the version in the header.

[bjbvoesenek]

Wow that's awesome! So BioRad also works with the website?

Thanks!

[ifokkema]

Wow that's awesome! So BioRad also works with the website?

Good that you asked... I double-checked; this branch doesn't have that code, I'll fix it.

[ifokkema]

Wow that's awesome! So BioRad also works with the website?

It works now with the Bio-Rad data as well! I had to merge release/2.0, with the Bio-Rad code, into this branch, then I put this branch online. So now all the new code is in this branch. When we're done, this branch can be merged back into release/2.0 (this PR), and then we can create a PR for merging release/2.0 into main. That will be the final check for 2.0.

[bjbvoesenek]

I modified the manual of Maurice:

• Removed the 'lab work' part
• Information on how to export data from the LightCycler 480 is still included
• Instructions for transpose and LinRegPCR are still included
• Adapted naming of the sheets in Excel
• Removed N0 calculations (not performed in our script)

If you agree, we can merge

[ifokkema]

So you don't want to put that information in our readme as we discussed? You want to keep the PDF for instructions, so users still have to switch from the readme to the PDF and back? If so, the readme needs to be modified because it refers to pages 12 - 14 in the PDF. But, I really recommend just copying the text over into the readme and using the images from the PDF. You mentioned you extracted them successfully, right?

Either way, I also recommend removing your last three commits and retrying uploading the PDF. Every time you commit a file, it gets stored and kept in Git forever. The removal, uploading, removal, uploading; it creates copies of that PDF internally. For text files, it's not that bad, but a binary PDF is much larger. Just remove the last three commits and upload the correct file. The push to GitHub will then need to be forced to overwrite the last three commits. If you don't know how, I can do it for you.

(by the way, you can replace a binary file in one commit, but that's just a detail)

[bjbvoesenek]

Done! Shall we then remove the qPCR manual or keep it for background information?

[ifokkema]

Done! Shall we then remove the qPCR manual or keep it for background information?

I'll leave that decision up to you! I'm not qualified enough in this field to properly assess its usefulness.

Just let me know when you're done, and I'll make one more pass to check everything.

[bjbvoesenek]

Okay, than I would suggest to keep it. Otherwise, we have to put the formulas in the readme.

So, I think we are ready!