Review Initial Partial Draft LBM

[peter-michalski]

Please review the structure of the initial draft c76b7e0eedeee44ddf6a8eedfb6516df8a870643

I have also completed the first draft of the following sections, in case you are interested in reviewing these now:

• Domain Analysis
• Method
• Results

The report is already fairly long. Might need to remove/cut down some sections or Appendices. Let me know what you think.

[smiths]

I apologize for the long delay in getting back to you @peter-michalski. I only have 5 minutes right now for review, but I will get back to this task soon.

• The appearance of the document and its organization looks good
• I'm impressed that you've made so much progress.
• Section 5.5.1 is the sole sub-section for Section 5.5. Usually if there is only one subsection, then the subsection doesn't need to be a subsection. The content of the subsection can usually be elevated to the parent section, or the content of the parent section can be used to form a second subsection
• Reference material should be in a chapter of its own. It should be part of the front-matter. Your report should start with Chapter 1
• The introduction (in what is currently Chapter 2) doesn't say enough about the Lattice Boltzmann method. What is the big picture motivation for the method. What problems is LBM suited to. What is its advantage over other methods for solving PDEs? A picture would be nice here.

I'll give more detailed comments soon.

[smiths]

Feedback up to and including Chapter 2 can be found in the following document:

StateOfPractice/Peter-Notes/Report/ReportFeedback.pdf

Updated in commit 32f37bb..

[smiths]

Feedback on Chapter 3 is given in 027a65f.

[smiths]

Feedback on Chapter 4 is given in fe7304e. We should think about removing some parts of the commonality analysis. It is probably best to focus on parameters of variation that can be expressed as values of specific types.

[smiths]

@peter-michalski, I've been thinking further about Chapter 2 (Domain Analysis). Please explain in the introduction chapter and in the intro to Chapter 2 the purpose of the commonality analysis. We added the commonality analysis to our methodology because the first time we did the State of Practice exercise we found we weren't comparing "apples" to "apples." If you view our ranking exercise as a means for someone to select the right tool for their job, then functionality is important. If they need a 3D Lattice Boltzmann solver, a 2D one (no matter how good), will not do the trick. Therefore, we want to classify the different software products based on their functionality.

The emphasis on functionality is important. This emphasis means that you can leave out the nonfunctional aspects in your CA. We can say that the nonfunctional qualities are assessed through other means in our process. For instance (even though we aren't going to do it this time), the methodology has a section on usability. These measures of usability are where we assess this nonfunctional requirement, so you don't need to mention it in the CA.

[smiths]

Feedback on Chapter 3 (Methodology) is available in 8d476de.

[smiths]

I've thought further about Chapter 3. I think we need more material to explain the methodology for measuring each software package with our large spreadsheet. You have the details in the Appendix, but the reader isn't likely going to read the appendix. Some examples of the tasks used for measuring would be nice. You don't need the complete template, but some examples would be good. Example interview questions would be good as well.

[smiths]

Feedback on Chapter 4 is given in f52e915.

[peter-michalski]

@peter-michalski, I've been thinking further about Chapter 2 (Domain Analysis). Please explain in the introduction chapter and in the intro to Chapter 2 the purpose of the commonality analysis. We added the commonality analysis to our methodology because the first time we did the State of Practice exercise we found we weren't comparing "apples" to "apples." If you view our ranking exercise as a means for someone to select the right tool for their job, then functionality is important. If they need a 3D Lattice Boltzmann solver, a 2D one (no matter how good), will not do the trick. Therefore, we want to classify the different software products based on their functionality.

The emphasis on functionality is important. This emphasis means that you can leave out the nonfunctional aspects in your CA. We can say that the nonfunctional qualities are assessed through other means in our process. For instance (even though we aren't going to do it this time), the methodology has a section on usability. These measures of usability are where we assess this nonfunctional requirement, so you don't need to mention it in the CA.

@smiths I addressed this issue in d56d91e984b6a06b5646307324564b9c98737a45

[smiths]

I had a quick look at the changes @peter-michalski. You are on the right track. I'll give the changes a more detailed review when I re-read the entire document.

[peter-michalski]

Feedback on Chapter 3 (Methodology) is available in 8d476de.

@smiths applied these changes in 23b372b087728f69c7cb30d634ccdd802e12ae5d.

Page 43 of the chapter 3 feedback pdf suggests putting the research questions into the introduction. Page 32 of the same pdf suggests putting a reference to the questions into the introduction.

I have kept a reference to the questions in the introduction. The questions take up an entire page and I think they are better placed in the Appendix.

[smiths]

The question define the entire purpose of what you are doing. They should be in the introduction. You can't really introduce a report and then tell the reader to read the Appendix to see the questions that the report is answering. If you are worried that the questions are too long, then you can look for a way to phrase them more succinctly, or integrate them into your presentation of the material. I really think the questions should be given in the introduction, not just a reference to the questions.

[peter-michalski]

The question define the entire purpose of what you are doing. They should be in the introduction. You can't really introduce a report and then tell the reader to read the Appendix to see the questions that the report is answering. If you are worried that the questions are too long, then you can look for a way to phrase them more succinctly, or integrate them into your presentation of the material. I really think the questions should be given in the introduction, not just a reference to the questions.

Good point. I will add the questions into the Purpose subsection of the Introduction.

[peter-michalski]

@smiths Moved the research questions to the introduction in 36c5ad04da19943798406c23c281fe54b40f7de0

[peter-michalski]

Feedback on Chapter 4 is given in f52e915.

@smiths these changes are applied in 3d923a3f011b8a211be0ce1113b5900b7cee2d57

A couple of points:

1. In Report_Chapt4_Feedback.pdf, page 49, I'm not sure what is written near the bottom right. "What was observed from the ????? (survey?)".

2. I will need more information/feedback about another comment on page 49. The left hand side comment regarding "Mondeley". How do we go about doing this? Is this an idea to consider or something we definitely will want to do?

[smiths]

1. "What was observed from the initial survey."
2. Mendeley isn't something we need to worry about right now. When we are done we should put the data on-line, so that others can inspect and judge our complete results. It isn't difficult to do. Here is an example of the data from one of the previous studies: https://data.mendeley.com/datasets/67ncspgz8n/1

[peter-michalski]

1. "What was observed from the initial survey."

2. Mendeley isn't something we need to worry about right now.  When we are done we should put the data on-line, so that others can inspect and judge our complete results.  It isn't difficult to do.  Here is an example of the data from one of the previous studies:
   https://data.mendeley.com/datasets/67ncspgz8n/1

@smiths Thank you. Fixed typos in 4e3895faf3dfebaff403b07d5bfeca27903d38d7

[smiths]

Chapter 5 is reviewed in the pdf pushed in commit 3e07dc3. @peter-michalski, don't let all of the marks on the paper discourage you - I really liked this chapter. I've always been optimistic about a paper, but now I really think we have some valuable things to say. :smile: @Ao99, if you are wondering about the kind of recommendations you can make to the medical imaging community, the content in the Peter's Chapter 5 would be helpful to you.

In a future SOP meeting, I think we should do the following as a group:

• go through identified pain points as a group and think about what advice we can offer (I wrote some ideas on the pdf, but my guess is that @JacquesCarette will have additional ideas that will come out with some group brainstorming.)
• go through qualities and how we can make things better (it is easy to just write a bunch of Parnas inspired words like information hiding and abstraction, but it would be nice if we could make our advice more domain specific.)
• design review of LBM (and medical imaging) software - can we see some specific problems that the developers can address? I'm wondering if looking at their code will suggest specific ideas on ways to improve design in these software domains. It might be too difficult to do this without domain knowledge, but there also might be some "low hanging fruit" that we can point to. @JacquesCarette has a real ability for detecting "code smells". :smile:

The items listed above aren't critical for writing your report @peter-michalski. I'm thinking about our paper.

[peter-michalski]

@smiths Thank you for your review of Chapter 5. Regarding Section 5.6, I was thinking that "research community" could also refer to rankings by our domain expert. Do you think this would be appropriate? I agree we could also add a comparison of number of publication citations (of those publications that are listed on the package webpages), and number of stars (of those that are on GitHub/GitLab) - although age of the project may play a role in this measure (we could comment on this if we go this route).

[JacquesCarette]

So should there be a (separate) issue for me to look at particular LBM/medical imaging software, with an eye to find actionable improvements to the software?

[smiths]

This is a great offer @JacquesCarette. I think it would be worth a few minutes of your time to look at some existing domain specific software. @peter-michalski, could you create an issue for @JacquesCarette to do a "smell test" on some LBM software? In the issue, please give links to specific code for @JacquesCarette to look at. Maybe we should try two examples: one that is at the top of our ranking, and one that is in the middle of the ranking. I feel like it won't be that meaningful to critique code that is at the bottom of our ranking.

[peter-michalski]

@smiths @JacquesCarette Issue created #80

[peter-michalski]

@smiths You commented on page 82 of ReportChapt5_Feedback.pdf to point to a template of your SRS. My initial thought was this template: https://gitlab.cas.mcmaster.ca/smiths/cas741/-/blob/master/BlankProjectTemplate/docs/SRS/SRS.pdf

Since it's not public, did you want me to add the SRS to AIMSS?

Let me know if you are referring to something else.

[smiths]

Hi @peter-michalski. I think the cas741 course notes are publicly available, at least that is how I meant to set them. :smile: I'm not thinking of something as specific as the 741 notes though. I'm thinking of my publications on the topic of requirements templates. From the pub References.bib file we have the following papers: SmithEtAl2007, SmithAndLai2005 and SmithEtAl2019_arXiv.

[peter-michalski]

Hi @peter-michalski. I think the cas741 course notes are publicly available, at least that is how I meant to set them. smile I'm not thinking of something as specific as the 741 notes though. I'm thinking of my publications on the topic of requirements templates. From the pub References.bib file we have the following papers: SmithEtAl2007, SmithAndLai2005 and SmithEtAl2019_arXiv.

Thank you

[peter-michalski]

Chapter 5 is reviewed in the pdf pushed in commit 3e07dc3.

@smiths the changes are reflected in 90a8d02fa7e36319097064da357238f49bb9981b.

A few notes:

1. Regarding the top comment on page 80 of the feedback file, some things have been moved around regarding this point, paragraphs are now generally split along pain points. Do you prefer a different format than paragraph text? itemizing?
2. Top comment page 82: Do we want to make this a checklist in my M.Eng. Report, or something to consider for a paper?
3. Bottom comment on page 82 (Smith and Carette paper): The comment addresses productivity. Not sure if this fits in here. Where to put this in the report? Time constraints for new features are mentioned earlier, but does this paper translate to what is considered by the developers here?

When you get a change please review the report again, if you wish to do so. I will finish up the Conclusion and Abstract soon. Hopefully I can get it to @JacquesCarette for a review soon.

[peter-michalski]

@smiths The abstract and conclusion have been added in 62010f99ecf5a43381d0abc63718cf5024a5d2c8.

[smiths]

@peter-michalski, I've reviewed the complete document, as given in the marked up version in 1beb0df.

Looks great! I've made some comments that should be addressed, but this is really coming together nicely. I'm optimistic that pulling a paper out of your report shouldn't be too difficult.

When you have addressed my comments, please close this issue and create a new one for @JacquesCarette to review your report.

[peter-michalski]

@smiths thank you for the feedback. It is incorporated in 6599cd7cdf3cc33057ad218014a19b97149b52b3.

I had a chance to review Reza's recommendation (https://geb.sts.nt.uni-siegen.de/doxy/musubi/index.html) and although the package is not on GitHub/GitLab it could fit criteria for measurement. It might not have made some authoritative lists because it is not on GitHub, but that did not stop some packages from making other such lists. We will need to decide if we want to proceed with adding it to a paper, or if we want to omit it (with comment) since it was not on our initial authoritative lists, and because any measurements taken now will be compared to measurements that are more than a year old.

I have created issue #84 for @JacquesCarette to review the report.

[smiths]

Thank you for the quick revisions @peter-michalski.

With respect to Reza's recommended software, we definitely don't need it for your report. However, you should include a note that the software came to your attention, but too late to be included in the analysis

I don't think we'll be able to get out of adding the software for the paper, but we can discuss this with @JacquesCarette. I updated a previous SOP analysis a year or so after the fact for one of our previous papers. It isn't as bad as it seems, since most things don't change much.

[peter-michalski]

@smiths I have added a note about Musubi software in 68d0ce9078fb080fe2d3ef01cd137d323e830ec2 and bca123ff127abc516c70e9807c51dc491f4d118e

[smiths]

Great!