search

maxheld83 / muggle

Opinionated Devops for R Data Products Strictly Without Magic
https://maxheld.de/muggle/
Other
13 stars 2 forks source link

submit talk to useR 2021 #192

Closed maxheld83 closed 3 years ago

maxheld83 commented 3 years ago

Between GUI-based reports and scripted data science lies an unproductive valley that combines the worst of both worlds: poor scaleability and high overhead. To avoid getting stuck there, small and medium-sized teams must 1) build strategic data products (not one-off scripts), 2) adopt software development best practices (not hacks) and 3) concentrate on business value (not infrastructure).

1) Strategic data products focus on the ETL pipelines, common visualisations and other modules that are central to the mission. These unix-style building blocks can then be recombined into various reports. 2) These modules are designed "as-if-for-CRAN" and written as type/length-stable, unit-tested and exported functions. 3) If something is not related to our mission, we rely on industry standards (Docker) and CaaS/DBaaS (Azure, GCP).

{muggle}'s opininated DevOps provides some technical scaffolding to help with this transition. It standardises the compute environment in development, testing and deployment on a multi-stage Dockerfile with ONBUILD triggers for lightweight target images and leverages public cloud services (RSPM, GitHub Actions, GitHub Packages). In contrast to some existing approaches, {muggle} never infers developer intent and has a minimal git footprint.

Success also requires a cultural shift. Development may still be agile, but it must not build prototype code. Fancy plots and reports are good, but reproducibility is more important.

We believe this is a necessary change to ensure value generation, and thereby, to ensure the future of democratic, and open-source data science.

maxheld83 commented 3 years ago

@njahn82 I decided spontaneously to submit muggle to useR (which will be remote again, conveniently). Since this is largely based on our experiences / practices at SUB, can I list you as a co-author?

njahn82 commented 3 years ago

Cool, thanks!

On Fri, 26 Mar 2021 at 09:55, Max Held @.***> wrote:

@njahn82 https://github.com/njahn82 I decided spontaneously to submit muggle to useR (which will be remote again, conveniently). Since this is largely based on our experiences / practices at SUB, can I list you as a co-author?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/subugoe/muggle/issues/192#issuecomment-808049162, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM7YRX5IMVHYUA3MLFNJ4TTFRDYFANCNFSM4ZT72FSQ .

maxheld83 commented 3 years ago

this has been accepted.

Message from Chairs to Authors:

Screenshot 2021-05-11 at 09 26 42

(submission 284 is shinycaas, as per https://github.com/subugoe/shinycaas/issues/60)

Should note that shinycaas has gotten better feedback, so I should probably pivot the presentation a bit.