Python HTML report generator

colinjennings commented 4 months ago

@colinjennings , which of these items have been done since our meeting yesterday?

[x] Move Colin's files to his own folder
[x] Upload Colin's code to his personal GitHub and submit a pull request to this GitHub
[x] Add percentage labels on pie chart
[x] Remove labels and strings off pies that are <10 %
[x] At the bottom of the html report, have hyperlinked granularity of folders of the biggest 5 parent folders
[x] After a hyperlink is clicked, move focus to the rendered table so the user do not wonder about if anything has happened (optional, can leave if too hard)
[x] Tashrif will fix mailreport.sh
[x] Remove the HH:mm timestamp from wherever needed in the html report (must remove :SS from all timestamps)
[x] Next day, Colin will demo how his program works when there are multiple dirs in _config/dirs.txt
[x] Add a variable month/year to look from, default should be last 12 months
[x] Follow color scheme of report-20231224.html as reference
[x] Change to only month/day such as 5/18 - 5/25:

1 Week Changes for /data/pnl/ (2024-05-18 - 2024-05-25)
[x] Round all GB sizes to two decimal points only
[x] Can you make the percentage label transparent or smaller? As you see, one is eclipsing the other now.
[x] Can you set it up in a way that after the expanded directory is clicked again, the table disappears from the view?

tashrifbillah commented 4 months ago

Primary problem of this work is that matlplotlib is not properly set up. Hence, the feature is not working without the availability of GUI. It worked last time because we were sitting at our computer. But cron will not have any GUI:

**PuTTY X11 proxy: unable to connect to forwarded X server: Network error: Connection refused
qt.qpa.xcb: could not connect to display localhost:10.0
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, minimal, minimalegl, offscreen, vnc, webgl, xcb.

Aborted (core dumped)

Edit: Fixed by: https://github.com/pnlbwh/diskusage-logging/commit/ab62e5391ad5c35b9c6b1fae59e58c85603e1e30

tashrifbillah commented 4 months ago

You can follow my code to fix the issue:

https://github.com/pnlbwh/TBSS/blob/2fb1bc776b07013b450c24057db876486fc4bb2b/lib/writeHtml.py#L23-L26

Edit: Fixed by: https://github.com/pnlbwh/diskusage-logging/commit/ab62e5391ad5c35b9c6b1fae59e58c85603e1e30

tashrifbillah commented 4 months ago

TODO for Tashrif and Colin--

[x] Update the report-${datestamp}.html name to report-rfanfs-${datestamp}.html and report-data-${datestamp}.html.
[x] Also update the subject in https://github.com/pnlbwh/diskusage-logging/blob/17596017c4da58bc29ba5e922d629cf28ae4e98e/mailreport.sh#L17

tashrifbillah commented 4 months ago

Thank you for addressing the comments so far. I am waiting for weekend's /data/ run to complete. /rfanfs/ run completed successfully though.

tashrifbillah commented 4 months ago

Colin, /data/ report just came out. Here are the issues we noticed:

Can you make the percentage label transparent or smaller? As you see, one is eclipsing the other now.
Can you set it up in a way that after the expanded directory is clicked again, the table disappears from the view?

colinjennings commented 4 months ago

Would it be acceptable to nudge the bottom label down if it is going to collide with the top label?

tashrifbillah commented 4 months ago

How would you know when it is going to collide upward or downward?

tashrifbillah commented 4 months ago

Colin, please put the latest graph screenshot here for record. The one that you sent me in Teams.

colinjennings commented 4 months ago

colinjennings commented 4 months ago

[x] Move size increase column ~~- [x] Convert to percent~~
[x] Only display directory changes 3 layers deep
[x] Do not show things with no size increase

tashrifbillah commented 4 months ago

Colin, I have deployed your latest commit. So it is going through another round of testing this weekend. Also, update the above checklist if appropriate. I feel like you completed the last item too.

tashrifbillah commented 3 months ago

Either way, I think you can undo the two commits you made today morning.

tashrifbillah commented 3 months ago

I confirm that the Friday evening's commit solved the issue of showing parent and children directories under change sections. Now the directories are unique.

colinjennings commented 3 months ago

[x] Sentence case where applicable
[x] Left-shift titles
[x] Make sure full directories are spelled out
[x] Align tables

tashrifbillah commented 3 months ago

New TODOs:

[ ] solve the bug at https://github.com/colinjennings/diskusage-logging/blob/d0049b70c7b5873a008a792d9c0e02e8e487b0be/html-report-generator/get_biggest_changes.py#L33
[ ] add a requirements.txt at root level with three packages: jinja2, matplotlib, pandas
[ ] revise installation instruction with something like pip install -r requirements.txt instead of all the R stuff. Colin can have liberty in this regard.

colinjennings commented 3 months ago

could you please clarify the nature of the bug at https://github.com/colinjennings/diskusage-logging/blob/d0049b70c7b5873a008a792d9c0e02e8e487b0be/html-report-generator/get_biggest_changes.py#L33

tashrifbillah commented 3 months ago

Colin, we can wait until your access to email is restored. I wrote everything in email.

tashrifbillah commented 3 months ago

Colin, this is the checklist you need to work on here or privately on every commit:

- [ ] Pushed up commit
- [ ] Downloaded code to /tmp/
- [ ] copy /to/official/location
- [ ] Run generate_usage_report.py and confirm success
- [ ] Repeat steps 2-4 in both rfanfs and data

colinjennings commented 3 months ago

[x] Pushed up commit
[x] Downloaded code to /tmp/
[x] copy /to/official/location
[x] Run generate_usage_report.py and confirm success
[x] Repeat steps 2-4 in both rfanfs and data
[x] Update installation instructions
[x] Remove time string from title

tashrifbillah commented 3 months ago

Hi Colin, can you remove the 05:57 part? This info is redundant for the scope of this report.

tashrifbillah commented 3 months ago

Colin, it appears that you did not deploy this change. Just check the report you got in email and you will see that the time string still exists.

colinjennings commented 3 months ago

Colin, it appears that you did not deploy this change. Just check the report you got in email and you will see that the time string still exists.

My mistake Tashrif -- just deployed the changes.

tashrifbillah commented 2 months ago

Thank you Colin. The report looks good now. I plan to find a time to sit with you and complete code review. We have reviewed output so far.

Also, I would like to review the installation instruction with you.

tashrifbillah commented 2 months ago

Hi Colin, I would like to merge your PR today.

pnlbwh / diskusage-logging

Python HTML report generator #13