Full stack dev: topics to teach

jph00 commented 3 years ago

This is the early planning of a new course "uncool full stack development" or something similar. Basically, I want to show people how to use simple broad-purpose tools, rather than SAAS and complex stuff like docker. So we'll be looking at cron, bash, perl, simple VPS servers, etc.

Why? Because: it's cheaper (use a single cheap server to do loads of stuff, rather than signing up for heaps of SAAS that do one little thing); you learn the foundations of everything else, which is knowledge that never goes out of date; rather than learning lots of domain specific languages, config files, and APIs, for every different service, you learn a small number of powerful foundations you can re-use everywhere; when you're working with SAAS or higher-level frameworks, you can debug your problems because you know what's happening underneath; lots of interviews ask about lower-level foundations, so it'll help you get recruited; for building a startup or hobby project, it's far cheaper, easier, and more scalable when you're in control of what you're doing and you understand how it all works.

Contents (links to sections)

Other similar syllabi:

Startup Engineering by Balaji S. Srinivasan and Vijay Pande; notes on first few lectures
CS50 and CS50W
https://wizardzines.com (we can use these in the course, if we link to the relevant page on wizardzines.com for any of the images we show in the video. For example https://wizardzines.com/comics/bash-tricks/)

jph00 commented 3 years ago

CLI tools

perl: -p -i -n -e -l -a
git / gh
apt / apt-fast
sudo
ack
cron
ssh & tunneling
shell
- scripts / #!
- pipes
- variables, variable substitution; !! , !$
- pushd / popd / cd -
- bashrc
- completions
- subshells; source
npm (global vs local)
jq (e.g automate pypi/github api)
parallel (e.g git_pull_all)
rsync (local; ssh; delete)
tmux (as nohup; panes)
vnc

jph00 commented 3 years ago

Concepts

Public/private keys (ssh; https; dkim)
IPs/subnets/ports (and secgroups)
DNS / coredns / dig / PTR
- Secondary: zoneedit / puck.nether.net / buddydns
fork / prefork / async
reverse proxy / fastcgi / wsgi / asgi
Env vars / PATH
VPNs / wireguard

jph00 commented 3 years ago

Protocols/foundations

TCP / UDP / unix sockets
HTTP
SMTP
SSL / letsencrypt
Web scraping (selenium etc)
supervisor/systemd
Package management (conda, apt, pip)
Documentation
openstack / aws
Cloud block storage
CRUD
Rest
Authentication
Server setup, security, and firewalls
Deployment: standalone; web-based
Automation

jph00 commented 3 years ago

Programming tools

Python
regex
- Use to find stuff in web page, e.g cheap TLDs at https://www.sav.com/domains, using Chrome regex search
glob
HTTP/ASGI servers, cookies (same-site)
HTML
CSS: grid, flexbox, ?tailwind
JS / Typescript: modern JS without libs
Editors: vscode; vim; emmet
?golang ?julia (implement protocols e.g POP3, SMTP, DNS)
sqlite (other DBs too?)
SQL
Debugging and debuggers
Profiling
Testing (unit / integration / continuous)
REPLs and notebooks
Chrome dev tools
Simple queues

jph00 commented 3 years ago

Basic services

Stripe
Mail delivery (AWS SES; elasticemail; sendpulse); mail-tester; Postmark DMARC; DKIM validator
VPS (DreamCompute; OVH; EthernetServers; Vultr; Genesis Hosting; Kamatera; ssdnodes; AWS LightSail; DigitalOcean; vpsdime)
- VPS Benchmarks and LowEndBox
- Openstack / AWS CLI / etc
- snapshots
- load balancing, DNS round robin (NB: CloudFlare with proxy effectively low TTL), host redirect
Cloudflare
GitHub: Actions; Codespaces; Issues/PRs; Releases
Building your own server

jph00 commented 3 years ago

Things to implement

Best is projects in your subject matter domain
- Or reach out to subject matter experts and offer to help them (e.g. data journalists; non-profits; clubs...)
- Make sure if you build something for someone else, you're willing to maintain it as long as it's needed!
zapier stuff
fut.io
guerrillamail.com; develmail.com
status monitor
- email; dns; RAM; uptime; df
AWS services?
wireguard VPN (access your home PC from anywhere)
pinboard.in
pagerduty
google analytics
RSS reader
twitter thread
basic web / tcp server
turn a notebook into a web app

jph00 commented 3 years ago

Startup skills

MVP (smallest complete thing people would love)
Product/market fit
What's it like to be in a startup
Iterations
Branding (incl Google search; Twitter cards; Analytics; Rachel's post)
Support
Book-keeping
Build vs buy
Scaling
Hiring
Funding
Doing good, and avoiding harm (including to yourself, your staff, your users, society...)
Creating a company (when/why?)
Competitive analysis
Pricing
Things to not waste time on

jph00 commented 3 years ago

Little VPS projects

Full list

VPN
Web crawling
Domain/service pings...
Find deals on #lowendbox
"a DokuWiki for keeping my thoughts organised. Because it has permission access control, I can make user accounts for friends/families/lab mates for specific categories so we can share data"
https://expiring.link/ (made in a weekend)
cleaver.cloud imitation
https://coronavirus.dunderdata.com/
backup storage
web ebook library
Minecraft server
video transcoder
personal SVOD (“Netflix”)
intrusion detection
video streaming + chat room
web server for staging/testing
Apache Superset dashboards
"a cron that sends me a reminder email about work emails that I should've responded to - but didn't" ("created model that "detect" emails I should care about (Uvicorn server hosted on AWS); created a Node app based outlook add-in that checks with the model and saves the email id/subject; node-cron send a reminder every other day")
pi.hole
receive/act on stats from home automation sensors, notifies if batteries run low, plants need to be watered...
"bots (Go and Ruby) are used to provide personal notification on Telegram and Discord, to quickly perform simple trades via API on crypto exchanges and to provide services (like price queries etc.) to crypto communities"
home automation and video server; object detection on video feeds to minimise false alerts
nextcloud
http://notify.run
"A Twitter bot that runs people's tweets on a 1980s home computer emulation and posts a video reply"
"https://graphly.snrrrub.com to track npm dependencies and associated open source licenses"
"https://nomly.snrrrub.com as a startup name generator (also a great drinking game)"
"I host http://aiquizzes.com on a $5 per month droplet"
"Habit tracking: Update streaks of habits on @todoist"
"Investing: Track currently open IPOs and add to to-do list on last day of deadline."
"Use @rescuetime API to send daily % change in time spent on social media"
"Fast Reverse Proxy so that I can tunnel into any of my machines"
"I have a server with a cron job that sends a small group of family and friends a daily cute animal picture. Uses PRAW to scrape the top entries of http://reddit.com/r/aww for the source image, downloads it, and then uses gmail API to send the email."
curl http://conda.ml gives you an install script for the latest Anaconda (Linux & Mac)
"Jenkins job that auto compiles a selection of papers/drafts in GitHub repo with papers and posts them to slack"
"http://RemoteOK.io is a single PHP file called "index.php" generating $65,651 this month. No frameworks. No libraries (except jQuery)"
"post a piece of content on your own server with your own URL. You do not have to host it externally."
proxy server to anonymize location information including VPN
TOR remote connection
"a cron job that checks for arXiv paper updates, downloads them to my Dropbox folder, and emails me so I remember to read the update"

radekosmulski commented 3 years ago

potential concepts to cover

how to copy/paste in a terminal
if I write a short script / program / little tool and want it available anywhere from the command line, where do I put it? (this falls under PATH but would be great to explicitly discuss this scenario)
- do such scripts need to be written in bash or can I use something else, for instance Python?
- how do I run something when my machine starts? (/etc/fstab for mounting, where do I put one off scripts I'd like run, understanding run levels and troubleshooting when the boot sequence doesn't complete)
how can I tell what's running on my system?
what are PIDs? how are they useful?
how is LOCALE important? how do I change keyboard layout (for typing in another language) without changing locale (setxkbmap)
mapping caps to ctrl - great for ergonomics, any other key mappings that are helpful?
how can I create an IOT device / remote sensor, what platform do I use?
if I create an API and expose it, can I let only authenticated users use it? how? how do I allow only my application to use the api (for instance, to communicate with my mobile app)
fail2ban
what is OpenStack, how can it be of use to us
in the context of a shell, what are the differences between interactive, non-interactive, login, non-login flavors and how do you arrive at each?
what is tty?
many technological stacks have interprocess communication, what it is and in what situations it is useful? what is the simplest way to implement it?
how do PPAs work? how do you add one? can you trust a PPA?
using a password manager, sharing the encrypted db within a team to facilitate work, what do you do with passwords when someone leaves the team?
file permissions, ownership
symlinking (hard, soft)
how do you decide what is worth to work on? how do you share work in a small team? is a situation where you have 3 equal parties even feasible or should the model always be one lead facilitating collaboration
ctrl-r to search for command in history
file permissions, file ownership
minimal css reset
{ margin: 0; padding: 0; box-sizing: border-box; }

section, article, aside, footer, header, nav { display: block; }


* nice intro video to [openstack networking](https://www.youtube.com/watch?v=yqFpyubsYfE&t=2s)
* resources for going deeper on certain HTML & CSS concepts:
  - the best source on the internet to learn HTML & CSS concepts, [internetingishard](https://www.internetingishard.com/)
  - [scrimba](https://scrimba.com/allcourses?price=free) - really nice intro courses to grid, flex, html & css
  - [really, really good game for learning and practicing the flexbox api](https://mastery.games/flexboxzombies/)
  - [superb blog on building websites](https://mastery.games/)
  - [fun little game on flexbox](https://flexboxfroggy.com) + [fun little grid game](https://cssgridgarden.com/)
* how to make our website accessible? that's a big one and a hard one probably, this is [a great resource](https://www.w3.org/TR/wai-aria-practices/) <- btw this also can work as a really good reference point if we are looking to implement some behavior with javascript, a lot of nice vanilla js there
* [placekitten](https://placekitten.com/) - very useful for rapid prototyping and experimenting
* two really useful commands for working with images on linux (making them smaller before adding to your site):
  - `mogrify -resize 960x500 <file_name>` - will resize maintaining aspect ratio to the longer side!
  - `identify <file_name>` - gives you the details on an image (dimensions, size)
- firefox grid inspector ([really nice quick overview of the tool](https://youtu.be/t0b3uBoDkBs?t=1394))
- [semantic-ui](https://semantic-ui.com/) - build websites using HTML, adding classes to style components, more elements than mvpcss
- [how one successful entrepreneur went about building his online businesses](https://twitter.com/searchbound/status/1374084768002891783?s=20), the [answer](https://twitter.com/searchbound/status/1374118857162522625?s=20) that he gave to my questions is also outstanding

AriMKatz commented 3 years ago

Hi @jph00

Stumbled upon this issue and glad you're taking a look at Julia.

Here are some resources that might be helpful in developing content:

In particular, the above blogposts describe Julia's philosophy behind its unique facilities for running external commands, and the issues with python, perl and bash that motivated it.

Also some fast and composable text processing libraries:

These are in addition to Julia's native regex capabilities.

Also, I know some individuals who are using Julia for web, sysadmin and general scripting/automation tasks and might be able to help with developing a potential Julia side of the course, I'll reach out to them.

Edit: Some web scraping libraries:

Gokkulnath commented 3 years ago

@jph00 This repo Link has a comprehensive list and could give you some information regarding various self-hosted (alternatives) tools available.

hamelsmu commented 3 years ago

One thing that I always struggle with is the topic of asynchronous programming, and dealing with promises and/or Futures.

I never feel like I truly understand how to use it correctly. I also have some kind of frustration whenever I'm forced to use it, because it seems to me that there is a fair amount of boilerplate required (but perhaps I just don't know the best way to do things). Just wanted to throw that out there

jph00 commented 3 years ago

One thing that I always struggle with is the topic of asynchronous programming, and dealing with promises and/or Futures.

I never feel like I truly understand how to use it correctly. I also have some kind of frustration whenever I'm forced to use it, because it seems to me that there is a fair amount of boilerplate required (but perhaps I just don't know the best way to do things). Just wanted to throw that out there

In what situations have you felt forced to use it, specifically?

hamelsmu commented 3 years ago

In what situations have you felt forced to use it, specifically?

Thanks for asking :). Situations where I have felt forced to use it:

Using javascript clients or libraries that have async functions in them for example octokit.rest.js.
When I'm using an API of some kind that takes a long time to return results, like machine learning as a service APIs. Sometimes I want to make many of these requests but don't want to block on their responses and not necessarily to use separate threads for that ( although thats what I often do out of laziness and because that is what I can more clearly reason about in my head).

P.S. one beautiful thing about fastai is allowed me to really understand the concept of callbacks more deeply, which are often entangled or a required way of thinking in async programming (at least with the programming languages and libraries I have tried to use).

dienhoa commented 3 years ago

This is a great topic ! I'm really keen on following your new course (or following how it is built). Can I ask where the repository of it ? Thank you for always helping people learning interesting things :)

gojira commented 3 years ago

Wow this is a fantastic endeavor! I have always wished that Startup Engineering could still be accessed. Here is another class you can check out - MIT's 'Missing Semester'

tcapelle commented 3 years ago

I came to also point to https://missing.csail.mit.edu/

hamelsmu commented 1 year ago

@seem @radekosmulski we should start teaching this course. Between us we have lots of stuff we have done already here.

seeM commented 1 year ago

@seeM @radekosmulski we should start teaching this course. Between us we have lots of stuff we have done already here.

Yes! I'm very excited about all of this stuff!

gojira commented 1 year ago

Love that you're picking this up!

I'd love to see coverage on app deployment / development. I'm thinking of stuff like the folloiwing that I'm having to do / learn about more and more without any real background.

TypeScript / node frameworks - Express etc.
Modify somebody's web app to what I need
Deploy on Vercel / Netlify

If somebody could distill the most valuable information about getting going with these in typical fastai fashion, I think it would be really valuable vs. spending 10 hours on some bootcamp with low information density.

jph00 / notes