Research contemporary JavaScript/React build toolchain

[hancush]

Background

Our current toolchain for transpiling and bundling JavaScript code, in a word, sucks. It seems like we struggle against it with every new project. it's not fun or a good use of development time to spend an afternoon debugging the same class of failures with every project.

Some contributors to the poor experience include:

• Lack of internal documentation of what each piece of the toolchain is, and is responsible for
• Templates make it easy to get started but tough to customize, because we lack the foundations gained from knowing how to set the tools up
• Indirection: Build commands are specified in a Django settings file, but work on inline and module scripts
• Inconsistency between projects, e.g., some use yarn, some use npm, some use different browserify / babelify plugins and options

Proposal

I propose we research alternative build toolchains that 1) transpile and 2) bundle our JavaScript code for use in the browser. The ideal toolchain would work independently of Django, so that it can be used for both Django/React and vanilla JavaScript projects. It should also represent an improvement in ergonomics over our current toolchain.

Deliverables

1. Brief survey of contemporary JavaScript build toolchains
2. Recommendation of adoption (if one is a better option than our current toolchain)
3. Clear, concise documentation of what each piece of the toolchain does and patterns for common configurations, e.g., specifying the location of files, formatting options, etc. (This is something we currently lack, that contributes to so many cycles spent puzzling over the current toolchain)

Timeline

2-4 investment days, depending on the tools and learning curve

[smcalilly]

we could maybe find something like this for django: https://github.com/rails/importmap-rails

[antidipyramid]

Looks like I've got the workplace app running using Webpack to bundle Javascript files. My initial thoughts on Webpack are that though there's a bit a learning curve, it's very well documented and has a very explicit configuration file, as opposed to Django Compressor.

The one nice thing about Django Compressor is that it's pretty much plug and play-- just install the package, set some config vars in settings.py and (in theory), you're good to go.

Local development with Webpack is best done with a development server that monitors your files and automatically recompiles bundles when they're changed. Of course you can forego the server but you'd have to manually run webpack every time you change a JS file. We'd run the webpack server locally as a docker-compose service. In production, we'd simply have webpack build the bundles as part of the deployment process (before python manage.py collectstatic).

You can find documented patterns for integrating Django and Webpack all over the web. Both Django and Webpack are so flexible in terms of configuration and file organization, so it's just a matter of fitting it into our current setups.

My next step is to try out the newer, more flashy Vite to compare with Webpack.

[antidipyramid]

I thought it would be more useful to demo webpack with a less complicated (in terms of logging in, etc.) app, so I made a webpack branch of EFI. Feel free to pull down and play around with it.

[derekeder]

@antidipyramid I tried setting up the EFI branch with webpack, but am getting an error when i build. Am I missing a step?

docker-compose down --volumes
docker-compose build

 => CANCELED [mpc-efi-webpack webpack 5/5] RUN npm install                                                                                                                              108.8s
 => [efi django 3/8] RUN mkdir /app                                                                                                                                                       0.3s
 => [efi django 4/8] WORKDIR /app                                                                                                                                                         0.0s
 => [efi django 5/8] COPY ./requirements.txt /app/requirements.txt                                                                                                                        0.0s
 => [efi django 6/8] RUN pip install --no-cache-dir -r requirements.txt                                                                                                                  68.0s
 => [efi django 7/8] COPY . /app                                                                                                                                                          0.7s
 => ERROR [efi django 8/8] RUN python manage.py collectstatic --noinput                                                                                                                   0.6s
------
 > [efi django 8/8] RUN python manage.py collectstatic --noinput:
#0 0.468 AWS config not found, defaulting to local storage
#0 0.468 Traceback (most recent call last):
#0 0.468   File "/app/manage.py", line 21, in <module>
#0 0.468     main()
#0 0.468   File "/app/manage.py", line 17, in main
#0 0.468     execute_from_command_line(sys.argv)
#0 0.468   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
#0 0.468     utility.execute()
#0 0.468   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
#0 0.468     self.fetch_command(subcommand).run_from_argv(self.argv)
#0 0.468   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
#0 0.468     self.execute(*args, **cmd_options)
#0 0.468   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
#0 0.468     output = self.handle(*args, **options)
#0 0.468   File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 187, in handle
#0 0.468     collected = self.collect()
#0 0.468   File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 105, in collect
#0 0.468     for path, storage in finder.list(self.ignore_patterns):
#0 0.468   File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/finders.py", line 130, in list
#0 0.469     for path in utils.get_files(storage, ignore_patterns):
#0 0.469   File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/utils.py", line 23, in get_files
#0 0.469     directories, files = storage.listdir(location)
#0 0.469   File "/usr/local/lib/python3.9/site-packages/django/core/files/storage.py", line 330, in listdir
#0 0.469     for entry in os.scandir(path):
#0 0.469 FileNotFoundError: [Errno 2] No such file or directory: '/app/assets'
------
failed to solve: executor failed running [/bin/sh -c python manage.py collectstatic --noinput]: exit code: 1

[derekeder]

@antidipyramid creating an empty assets folder did the trick and the project built for me.

However, after running it, the map does not load for me:

It would be good to open a PR with additional instructions on any steps to take to set this up. Though we would close it, having it as a reference would be helpful. https://github.com/datamade/mpc-efi/compare/main...webpack

[antidipyramid]

@derekeder Looks like the webpack server might not be up. Can you see if the webpack container starts up when you run docker-compose up?

[derekeder]

FYI shelling into the webpack container and running npm install resolve this. there must be something up with how the build is happening on the webpack container

[smcalilly]

@antidipyramid to write up step no. 4 in https://github.com/datamade/how-to/blob/main/CONTRIBUTING.md#4-recommend-adoption-further-research-or-abandonment

[antidipyramid]

Here's my proposal for Webpack adoption.

[smcalilly]

@antidipyramid to add some docs about it.

@smcalilly to set this up in the cookiecutter as the default toolchain.