DominiqueMakowski commented 5 years ago

What about submitting this to JOSS or Ropensci + JOSS?

strengejacke commented 5 years ago

I'm not familiar to Ropensci, but I have no objections against it. I would just give insight a lower priority than bayestestR, because bayestestR gives you results that you may directly refer to in a paper, and thus you could (should) cite that package, while insight is something that is below the "radar" of most users.

DominiqueMakowski commented 5 years ago

true, true, I understand.

strengejacke commented 5 years ago

Though I guess it's no big deal at least to prepare a submission to JOSS, because we have many points of the review-checklist already done, I think. We only have less content for a paper than we have for bayestestR. ;-)

strengejacke commented 5 years ago

I started preparing a draft for JOSS.

[x] The summary section needs some re-writing and probably some more content. I thought of something like "low lvl package for developes" and "no external pkg dependencies", as these are two core aspects of the package. Maybe we can add some text from the presentation I held at RUG Hamburg-
[x] Examples: I think we can add two or three working examples, and that should be OK then?

DominiqueMakowski commented 5 years ago

I agree!

DominiqueMakowski commented 5 years ago

and these examples could be re-used in a blogpost 😏

pdwaggoner commented 5 years ago

Hi guys - I know this is your core package and that I am new. But if you need any help on pulling together a manuscript for JOSS or in submitting to ROpenSci, don't hesitate to let me know. I would be happy to contribute. However, I would certainly be happy to not contribute here for the above reasons, or if you don't think it necessary. Given my membership, though, I felt like I should at least offer to help how/if I can and if there would be sufficient value added. Thanks!

strengejacke commented 5 years ago

Sure, feel free to contribute, if you like. I would say, depending on your input, we should decide about your role, either being a co-author or mentioned in the acknowledgements, or whatever. "Substantial" contributions should be valued by becoming a co-author.

My idea is to add some more content to the summary, maybe also a short sentence either on the goal / philosophy of the easystats projects as a whole, or the insight-package in particula, maybe based on this presentation:

https://github.com/easystats/easystats/blob/master/presentations/2019-02_Hamburg_RUG/1902%20Hamburg%20RUG%20easystats.pdf

pdwaggoner commented 5 years ago

Absolutely. I will thicken up the summary a bit via a pull request in the paper.md file. Feel free to list me wherever you feel is most appropriate, either as a coauthor or "contributor" to be acknowledged. I have no preference here; just wanting to help. Keep an eye out for the PR in a bit, with some summary ideas.

pdwaggoner commented 5 years ago

Just offered a draft JOSS manuscript. Take a look and let me know. Hope its helpful!

strengejacke commented 5 years ago

@DominiqueMakowski I checked the submission-ckeck-list for JOSS and I think we could submit the paper. Do you have any things to add / revise? One (among other?) points during the review could be the question regarding @pdwaggoner author's role, since you're not listed as contributor in the DESCRIPTION file. But since you made substantial contributions to the paper, I think you should be listed as co-authors. Let's see if reviewers are ok with it.

strengejacke commented 5 years ago

we probably need a version bump or change for the submission, I think 0.2.0.9000 is not suitable.

DominiqueMakowski commented 5 years ago

Great work guys!! Few comments:

Maybe we should consider changing the title of the paper. Something like "insight: A Unified Framework for R Models Information Extraction" looks IMO more powerful 💪
For the authors, I am sure the reviewers will be ok with Philip's presence :) As my role for insight was merely to help to find names for stuff, I think it would make sense to swap our positions so that Philip is second and I third.
Maybe we should change the phrasing in the models supported section to leave it a bit more open, so it remains compatible when we will add support for other models.
It would be nice to have some figures. I am not sure about the logo (although why not?), but maybe something to help to visualise the functions... I'll try to think of something.

I'll make a PR with a proposition of changes, feel free to modify ;)

strengejacke commented 5 years ago

https://joss.theoj.org/papers/30851fde6e76f138423f8f1fd9082831

strengejacke commented 5 years ago

I think JOSS does not scale images / figures. Thus, I decided to remove the first figure (package logo), because it would take too much space on the page and readers probably don't understand the reason. I hope you're ok with this last-minute-change, Dominique? (now sleeping, hehe)

DominiqueMakowski commented 5 years ago

not yet ;) Sure sure, go ahead!

strengejacke commented 5 years ago

pre-review has started: https://github.com/openjournals/joss-reviews/issues/1405 JOSS papers require a bibliography (which makes sense, of course), however, due to the "technical" nature of our package, we have no fancy statistical methods to cite. I have now cited the R-software, but if you have any idea if it's appropriate to add another reference, let me know here...

DominiqueMakowski commented 5 years ago

You preferred to keep the package's title as paper title?

strengejacke commented 5 years ago

You preferred to keep the package's title as paper title?

I merged your PR, so it should have changed to your suggestion... Or did something go wrong?

DominiqueMakowski commented 5 years ago

I didn't change the title though, as this was just an idea to discuss 😬 In any cases, the title is also good this way, however, JOSS would not mind if we decided to change it, even after submission (personal experience ^^)

strengejacke commented 5 years ago

Ups, sorry, I thought you already made the changes in the paper, I was ok with your suggestion... :-/

pdwaggoner commented 5 years ago

Yeah I too have changed JOSS titles after submission. Its pretty relaxed as far as journals go.

strengejacke commented 5 years ago

Just to track the review-repo

strengejacke commented 5 years ago

Dear co-authors, we got a first review on our JOSS-submission: https://github.com/openjournals/joss-reviews/issues/1412

I think some of the raised issues are quite useful in particular to make the package more accessible to users. I can start working on this by the end of next week earliest.

pdwaggoner commented 5 years ago

Thanks for this. The bulk of the conerns seems to be in the source code, which would likely be better handled by you guys as you wrote it. I would be happy to update the manuscript if/when that would be appropriate. Or do you think the paper needs to be updated? Would the changes to the source code be better addressed in a direct response to the reviewer? Let me know however best you see moving forward here. Obviously happy to do whatever is needed.

DominiqueMakowski commented 5 years ago

Let's do some brainstorming to share ideas on how to address the points and on how to improve the package. Here's what crossed my mind in no particular order:

API

Maybe model_info could indeed also wrap around all the other functions, which would become only shortcuts for specific stuff. The API would then become fairly easy:

info <- model_info(model)
info$data
info$parameters
info$terms
info$formula
...

Although not very efficient memory wise, hence the use (for developers) of the specific subfunctions., which would become an "advanced" level of usage.

testing: it feels like a loss of efficiency to organise tests by functions (as it would mean redundant fitting/download of models). Maybe we can find a workaround for it. The alternative way would be to reorganise the code by model types. This could be a little more difficult to maintain (as methods would be spread across files) but at the same time, it could provide a more comprehensive and unified view on what is available for each model.
rename find_algorithm. The reviewer suggested find_estimator... but maybe find_method would be more general in regards to what it does (for instance for Bayesian models).

README, DOCS etc

Common contributing guide in easystats and link to it from all packages
better diagram and conceptual explanation at the beginning of the readme, explaining the differences between parameters, terms, formula, predictors etc. A paragram like "What is a statistical model made of?" and then explaining the stuff
Emphasise only the get_* functions, and the find_* variants would only be mentioned in the get_* documentation as "shortcuts" to only extract the names
Update paper title

strengejacke commented 5 years ago

That are good points! But I think we don't need to address every point by changing something. According to the test/code organization: I think here we can argue why this structure makes sense.

mattansb commented 5 years ago

I've obviously not been involved in the dev of insigh, or the submission to JOSS, but please let me know if there's anything I can do, help, etc.

strengejacke commented 5 years ago

As I'm currently busy, I will come back to this earliest by the end of next week. Just some short ideas that came to my mind:

1) similar to the figure Dominique already created, I can think of another figure that contains a formula of a mixed model, like lmer(Reaction + Days + Weeks + (1 + Days | Subject), data = sleepstudy), which is also colored according to its parts, and we then connect the related functions to the formula-parts (find_response, find_predictor, find_terms, find_random, find_random_slopes). This probably makes clearer why we have different functions.

2) When we do this, we might see clearer if the function names really describe what they are referring to, and we may think of renaming functions.

3) We may also see clearer how certain functions like find_variables() or find_parameters() fit into this design philosophy.

4) I think we should do this for the paper, but maybe we should also take parts of our current paper and the above mentioned figure to make a vignette out of it. It's ok to have content twice, both in the paper and a vignette, I would say. By this, we would also address this issue: https://github.com/easystats/insight/issues/46 My impression is that describing the design of insight is - at least currently - more important than describing how to contribute code.

pdwaggoner commented 5 years ago

I agree that we can argue some stuff (I.e., with a cleaner discussion of things in the paper), instead of really changing much of the source code/package core. I like these ideas as well - I too am pretty busy the rest of the week but can get back to this Monday morning. Exciting!

pdwaggoner commented 5 years ago

I just went back to the JOSS review and it seems that the bulk of his concerns are with understanding how everything is pieced together in the package architecture. I think many of these points of confusion (e.g., differences between terms and formula, and so on), though reasonable, are now cleared up in the vignette @strengejacke just put together (referenced above). Additionally, its important to note that a JOSS paper is simply a high-level overview of a package to encourage users to jump into the package documentation for a closer look and for many examples. Here again, the vignette addresses more of this nuance.

Thus, it seems like there are really two realms of updates/edits that need to happen:

JOSS Paper: Here we really need to clean up maybe some of the language but not a ton. Rather, the big thing that we should do here in my opinion, is conclude the paper by pointing the reader/user to the vignette for a deeper look at the nuance of the package and the coding philosophy behind insight.
Package Documentation: Here is where we simply release the vignette with more detailed descriptions of things as @strengejacke has already done.

So my question is: for this paper, do you think it is sufficient to simply conclude by saying, "Go look at the vignette for more details", and then make sure the vignette is ready to go? Or should we do a more substantial overhaul of the package?

pdwaggoner commented 5 years ago

Hi all - just wanted to check back in on the JOSS submission? Didn't want it to fall through the cracks, especially since we got a great and quick review (seemingly pretty rare in the paper world).

I am happy to take the lead on making changes to the manuscript (after we agree on a path forward), and I think I should take a back seat on package changes. Let me know what you guys are thinking here as you have some time to get back to this. Thanks!

strengejacke commented 5 years ago

@pdwaggoner Yes, if you like, you can make the first steps! I'm happy to work on code revisions etc., but this week is deadline for a funding proposal, so I would start working / joining next week.

pdwaggoner commented 5 years ago

Sounds good. This is the first week of summer courses for me, so let’s plan to all revisit this next week. Sound alright? Thanks!

strengejacke commented 5 years ago

@pdwaggoner Are you going to start working on the review?

pdwaggoner commented 5 years ago

@strengejacke - absolutely. Its on my list for this afternoon. I am currently (over the next 2 months) in transition to the University of Chicago for a faculty appointment there. Thus, please excuse me if I am ever a little late on some things. I will try my best to continue to keep things moving on my end, but I wanted you guys to be in the loop. But yes, this afternoon, I am hoping to have a new draft of the JOSS manuscript and some point moving forward on updating code. Stand by...

strengejacke commented 5 years ago

Parts of the revisions also affect the readme in our GitHub repo, which we already revised. I think, both paper and GitHub-info files need to be revised in line, and then we can see what we should change in the code / docs as well. No worry when time is limited, contributing to the easystats project is voluntary and there is no "minimum pensum" anyone needs to reach here. ;-)

pdwaggoner commented 5 years ago

Sounds good and many thanks for the gracious response. I think I should be able to make some good progress on this today, though.

pdwaggoner commented 5 years ago

Alright @strengejacke and @DominiqueMakowski - I have gone through and significantly reworked the front end responding to most of the points raised by the reviewer, at least for all issues that could be addressed/defended in the paper as opposed to in the code. See a list of my changes in the comments associated with my most recent commit on the paper.md file in the paper folder.

Further, I am not sure I agree with a couple points: 1. cut back on functions, and 2. reduce the number of supported models to focus on GLMs. I think continued expansion of model support is the hallmark of the philosophy behind the package. So, with respect to the reviewer, I wonder if we can get by without addressing these issues? What do you think?

I think we are really close now with this draft. Take a close look and let me know what you guys think here. Exciting!

strengejacke commented 5 years ago

Thanks for the work on the paper, it really improved substantially. I have adapted some parts in order to revise the readme and the vignette, so all of these documents are better structured now (but not identical to the paper).

I think indeed this addresses some of the points raised by the reviewer. There are some suggestions in revising the code where I (at least partly) agree with. I try to address this within this week, though I would leave some aspects open for discussion when we resubmit our revision.

Also, I agree with your two points - we should not reduce functions, since find and get do something different. I think in revising the paper and readme, we made this distinction more clear, maybe this is already enough. And yes, we should not narrow our focus to models of particular families.

DominiqueMakowski commented 5 years ago

Nice updates! I agree with all the points mentioned above reducing functions or narrowing the scope.

I've also updated the paper title from:

insight: Easy Access to Model Information for Various Model Objects

to

insight: A Unified Syntax for Accessing Model Information for Various Model Objects in R

This adds two elements, 1) the "unified syntax" which is a powerful and meaningful keyword and 2) "in R" which makes the context clear from the paper's title. What do you think?

strengejacke commented 5 years ago

Maybe we can add a paragraph in the paper to address this point:

Examples: The documentation shows how a user can extract information from a model, but it doesn't show how the user can use this information to solve larger problems. Some applications showing how the various functions operate together (and how they are distinct) would be greatly beneficial.

I think we can mention packages like bayestestR, performance or ggeffects here, to show how other packages benefit from insight.

DominiqueMakowski commented 5 years ago

Indeed, accompanied by a nice self-citation 😁

DominiqueMakowski commented 5 years ago

and 2) "in R" which makes the context clear from the paper's title.

or maybe we should remove the "in R" to keep it more general? after all, I could see this syntax used and applied outside R...

strengejacke commented 5 years ago

Yes, but our package is in the R context, so I would keep it. JOSS has papers about different languages, and I think it's good if the title provides as many information as possible.

strengejacke commented 5 years ago

Do you think it makes sense to integrate the paragraph "Definition of Model Components" (and figures) into the paper as well?

https://easystats.github.io/insight/articles/insight.html#definition-of-model-components

DominiqueMakowski commented 5 years ago

I would tend to say yes

strengejacke commented 5 years ago

@pdwaggoner I have added a "short" paragraph (a bit longer due to the two figures) called Definition of Model Components and I am now adding a paragraph on Examples of use cases in R packages.

Once I have finished this, could you read over these passages and correct grammar, spelling, style where necessary? I'll comment here once I'm finished.

Also, I will link to the compiled PDF here, as I inserted some page breaks, so figures are visually clearer linked to the related paragraphs. Tell me, what you think (@DominiqueMakowski as well).

strengejacke commented 5 years ago

Ok, I finished my draft. Here's the PDF:

https://github.com/easystats/insight/blob/master/paper/paper.pdf

@DominiqueMakowski @pdwaggoner I would kindly ask you to take a look at the paper. We have now revised paper, code, and website/docs, so after you have looked at the latest paper draft, I would response to the reviewer.

pdwaggoner commented 5 years ago

This looks great! I went through line by line and cleaned up a lot of grammatical things and tried to streamline language (making it simpler) where ever possible. But none of the substantive content has changed.

These changes overall look really great and I think the paper is in good shape! I'd say send it back whenever you are ready. Great work all!

easystats / insight

Ropensci and JOSS #56

API

README, DOCS etc