search

entirelymagic / Programming-Cheat-Sheet

Programming Helper - Any kind of help is welcomed! Including and not limited to Python.
4 stars 2 forks source link
cheatsheet django django-rest-framework docker docker-compose git github linux python

readme

Partially covered:

1. General information

1.1. Create a VS Code Editor from web (press . o open when on main Repository page)

Bellow are example of how to create a vscode editor from web. IF you connect your account and synchronize it will load your settings. Open Online Editor Editor example

2. GIT

2.1. Cheat sheet

2.1.1. Local Changes

2.1.1.1. Create a git patch from the uncommitted changes in the current working directory

If you haven't yet committed the changes, then:

git diff > mypatch.patch But sometimes it happens that part of the stuff you're doing are new files that are untracked and won't be in your git diff output. So, one way to do a patch is to stage everything for a new commit (git add each file, or just git add .) but don't do the commit, and then:

git diff --cached > mypatch.patch Add the 'binary' option if you want to add binary files to the patch (e.g. mp3 files):

git diff --cached --binary > mypatch.patch You can later apply the patch:

git apply mypatch.patch

2.1.1.2. Display the status of modified files

git status

2.1.1.3. Add a file to staging as it looks right now

git add [file_name]

2.1.1.4. Add a folder to staging as it looks right now

git add [folder_name]

2.1.1.5. Commit staged files in a new commit

git commit -m "descriptive_message"

2.1.1.6. Add all files to staging and commit them at once

git commit -am "descriptive_message"

2.1.1.7. Unstage a file while retaining the changes

git reset [file_name]

2.1.1.8. Diff of what is changed but not staged

git diff

2.1.1.9. Diff of what has changed between staged changes and the last commit

git diff --staged

2.1.2. Branches

2.1.2.1. List all branches. The current one is marked with *

git branch

2.1.2.2. Create a new branch

git branch [branch_name]

2.1.2.3. Switch to a branch

git checkout [branch_name]

2.1.2.4. Create a new branch and switch to it

git checkout -b [branch_name]

2.1.2.5. Switch to the previously checked out branch

git checkout -

2.1.2.6. Rename a branch

git checkout -m [new_branch]

2.1.2.7. Delete a branch, locally

git branch -d [branch_name]

2.1.2.8. Merge another branch into the current one

git merge [branch_name]

2.1.3. Working with a Remote Repository

2.1.3.1. Fetch and merge all commits from the tracked remote branch

git pull

2.1.3.2. Fetch and merge all commits from a specific remote branch

git pull [alias] [branch_name]

2.1.3.3. Fetch recent changes from the tracked remote branch but don't merge them

git fetch

2.1.3.4. Push all local branch commits to the tracked remote branch

git push

2.1.3.5. Push all local branch commits to a specific remote branch

git push [alias] [branch_name]

2.1.3.6. Add a new remote repository with the given alias

git remote add [alias] [repo_url]

2.1.3.7. Display a list of remote repositories and their URLs

git remote -v

2.1.4. Commit History

2.1.4.1. Show all commits in the current branch’s history

git log

2.1.4.2. Show all commits in the current branch’s history by printing each commit on a single line

git log --oneline

2.1.4.3. Show number of commits per author on all branches, excluding merge commits

git shortlog -s -n --all --no-merges

2.1.4.4. Show number of commits per author on a branch, excluding merge commits

git shortlog -s -n [branch_name] --no-merges

2.1.4.5. Show number of commits per author on all branches, including merge commits

git shortlog -s -n --all

2.1.4.6. Show number of commits per author on a branch, including merge commits

git shortlog -s -n [branch_name]

2.1.5. Rebase

2.1.5.1. Reapply commits from the current branch on top of another base

git rebase [branch_name]

2.1.5.2. Abort a rebase

git rebase –-abort

2.1.5.3. Continue a rebase after resolving conflicts

git rebase –-continue

2.1.6. Undo

2.1.6.1. Revert the changes in a commit and record them in a new commit

git revert [commit]

2.1.6.2. Reset to a previous commit and preserve the changes made since [commit] as unstaged

git reset [commit]

2.1.6.3. Reset to a previous commit and discard the changes made since the [commit]

git reset --hard [commit]

2.1.7. Stash

2.1.7.1. Stash modified and staged changes

git stash

2.1.7.2. Stash modified and staged changes with a custom message

git stash push -m "message"

2.1.7.3. Stash a selected file by specifying a path

git stash push src/custom.css

2.1.7.4. List all stashed changesets

git stash list

2.1.7.5. Restore the most recently stashed changeset and delete it

git stash pop

2.1.7.6. Delete the most recently stashed changeset

git stash drop

2.1.8. Tags

2.1.8.1. Create a new tag

git tag "tagname"

2.1.8.2. List all tags

git tag

2.1.8.3. Delete a tag

git tag -d "tagname"

2.1.9. Repository Setup

2.1.9.1. Create an empty repository in the current folder

git init

2.1.9.2. Create an empty repository in a specific folder

git init [folder_name]

2.1.9.3. Clone a repository and add it to the current folder

git clone [repo_url]

2.1.9.4. Clone a repository to a specific folder

git clone [repo_url] [folder_name]

2.1.10. Global Config

2.1.10.1. Set the username

git config --global user.name "user_name"

2.1.10.2. Set the user email

git config --global user.email "user_email"

2.1.10.3. Set automatic command line coloring

git config --global color.ui auto

2.2. RE-clone

GIT=$(git rev-parse --show-toplevel)
cd $GIT/..
rm -rf $GIT
git clone ...

2.3. Clean and reset

git clean --force -d -x
git reset --hard

2.4. Clean

git clean --force -d -x

2.5. Reset

git reset --hard

2.6. Update local branch after main branch was renamed

git branch -m master develop

git fetch origin

git branch -u origin/develop develop

git remote set-head origin -a

2.7 Git LFS (Large File Storage)

Documentation: https://git-lfs.github.com/

Include in the current project files from the previews pushes of files that became larger then 100MB

This will include all csv files

git lfs migrate import --include="*.csv"

3. Python

3.1. Cheat Sheets

Link to the cheatsheet: Beginners Python Cheat Sheet

3.2. Data Types

3.2.1. Text Type: str

3.2.2. Numeric Types: int, float, complex

3.2.3. Sequence Types: list, tuple, range

3.2.3.1. List

3.2.3.1.1. Transform a nested list into a simple list
from functools import reduce

nested_list = [[1], [2], [3, 5, 6], [4, 3, 12, 33]]

simple_list = reduce(lambda x,y: x+y, nested_list)

3.2.4. Mapping Type: dict

3.2.5. Set Types: set, frozenset

3.2.6. Boolean Type: bool

3.2.7. Binary Types: bytes, bytearray, memoryview

3.3. Useful Functions

3.3.1. EAN generators

Generate EAN 13 barcode for products.

Formula used (5940000 + random_number + last digit validator)

EAN country codes : https://wholesgame.com/trade-info/ean-barcodes-country/

3.3.2. 3.2.2 XLSX Generators

3.4. Asynchronous Python

3.4.1. Asynchronous Web Socket Class

This is testws.py

!/usr/bin/env python3
# testws.py
import sys, json
import asyncio
from websockets import connect

class EchoWebsocket:
    async def __aenter__(self):
        self._conn = connect('wss://ws.binaryws.com/websockets/v3')
        self.websocket = await self._conn.__aenter__()        
        return self

    async def __aexit__(self, *args, **kwargs):
        await self._conn.__aexit__(*args, **kwargs)

    async def send(self, message):
        await self.websocket.send(message)

    async def receive(self):
        return await self.websocket.recv()

class mtest:
    def __init__(self):
        self.wws = EchoWebsocket()
        self.loop = asyncio.get_event_loop()

    def get_ticks(self):
        return self.loop.run_until_complete(self.__async__get_ticks())

    async def __async__get_ticks(self):
        async with self.wws as echo:
            await echo.send(json.dumps({'ticks_history': 'R_50', 'end': 'latest', 'count': 1}))
            return await echo.receive()

And this in main.py:

# main.py
from testws import *

a = mtest()

foo = a.get_ticks()
print (foo)

print ("async works like a charm!")

foo = a.get_ticks()
print (foo)

This is the output:

root@ubupc1:/home/dinocob# python3 test.py
{"count": 1, "end": "latest", "ticks_history": "R_50"}
async works like a charm!
{"count": 1, "end": "latest", "ticks_history": "R_50"}

4. Django

4.1. Cheat Sheet

Django Cheat Sheet

Djaneiro Cheat Sheet

Django Models Cheat Sheet

4.2. Django Extensions

This include also generation for database models using graphwiz.

https://github.com/django-extensions/django-extensions

4.3. Authentication

## Default Authentication

REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ],
    'DEFAULT_AUTHENTICATION_CLASSES': [ # new
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.TokenAuthentication', # new
], }

pipenv install dj-rest-auth==1.1.0

# config/settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # 3rd-party apps
    'rest_framework',
    'rest_framework.authtoken', # new
    'dj_rest_auth', # new

#config/urls.py

urlpatterns += [

    path('api-auth/', include('rest_framework.urls')),
    path('api/v1/dj-rest-auth/', include('dj_rest_auth.urls')), # new
]

4.4. User registration

pipenv install django-allauth

 config/settings.py
INSTALLED_APPS += [
    'django.contrib.sites', # new
    # 3rd-party apps
    'rest_framework.authtoken',
    'allauth', # new
    'allauth.account', # new
    'allauth.socialaccount', # new
    'dj_rest_auth.registration', # new
]

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' # new
SITE_ID = 1 # new

urlpatterns += [
    path('api/v1/dj-rest-auth/registration/', include('dj_rest_auth.registration.urls')), # new
]

4.5. Models

4.5.1. Implementing rules to Django models

4.5.1.1. Redirect

=======

4.6. Views

4.6.1. API views

4.6.1.1. Redirect

class ARedirectApiView(APIView):
  """ an example of APIView that take an optional parameter and redirect to the path provided"""

    def get(self, request, *args, **kwargs):
        """override get method for the APIView"""
        ean_code = self.request.query_params["ean_code"] # a random optional parameter

        if ean_code:
            image_location_in_media = "test.jpeg"
        else:
            image_location_in_media = "default_image.jpeg"

        host = request.get_host() # host is the domain name
        redirect_path = f"http://{host}/media/{image_location_in_media}"
        return HttpResponseRedirect(redirect_to=redirect_path)

4.6.1.2. Make a view accept single or multiple posts to database

This is an example for a view that accepts single(dict) or multiple(list of dict) post to the database.

  class ProductViewset(viewsets.ModelViewSet):
    """Product Viewset
    API endpoint that allows products to be edited.
    Allowed actions:
        "POST", "PUT"
    """

    queryset = models.Product.objects.all()
    serializer_class = serializers.ProductSerializer
    http_method_names = ['post', 'put']

    # Method that allow multiple or single product to be posted on database
    def create(self, request, *args, **kwargs):
        serializer = self.get_serializer(data=request.data, many=isinstance(request.data,list))
        serializer.is_valid(raise_exception=True)
        self.perform_create(serializer)
        headers = self.get_success_headers(serializer.data)
        return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)

4.7. Django Testing

4.7.1. Coverage

Installation

pip install coverage

Running and erasing coverage For running on a django project, use this command:

coverage run --source='.' manage.py test the-app-you-want-to-test

This command will fill a “.coverage”, located in COVERAGE_FILE and then you may see results or report. If you need to remove gathered data, execute: coverage erase

For a single file Maybe you only want to check a python code, then do:

coverage run your_program.py arg1 arg2 arg3

There are some additional options, take a look on https://coverage.readthedocs.io/en/coverage-4.3.4/cmd.html#execution

See results If you want to show the results in the command line, run:

coverage report

For more readable results run: coverage html

To know concretely what part of your code is covered by tests, use: coverage annotate -d directory-where-to-put-annotated-files

It will generate same source code file with an additional syntax on it:

Good coverage level A good coverage usually comes on 90%. However, if you see 100% it could be not so good signal because it could >be someone dealing with coverage instead of quality of tests.

Some tips:

4.8. Django 3.1 Async

4.9. Using with Docker

Here we will create a New django project inside a docker image. Can be extended to use with an existing project. Details will vbe added and further.

Create a requirements.txt file with the following content: ! this is the minimum architecture to run the project. We need Django and psycopg2


Django>=3.0,<4.0
psycopg2-binary>=2.8

Create a Dockerfile with the following content:

# syntax=docker/dockerfile:1
FROM python:3
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN pip install -r requirements.txt
COPY . /code/

Create a docker-compose.yml file:

Here you set Services: db:

web:

version: "3.9"

services:
  db:
    image: postgres
    volumes:
      - ./data/db:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    depends_on:
      - db

From the root of the directory:

Create the Django project by running the docker-compose run command as follows.

sudo docker-compose run web django-admin startproject composeexample .

After the docker-compose command completes, list the contents of your project.

ls -l

File list

5. SQL

6. Web

7. Operating Systems

7.1. Linux

7.1.1. Files and Navigating

7.1.1.1. Directory listening

ls

7.1.1.2. Formatted listening

ls -l

7.1.1.3. Formatted listening including hidden files

ls -la

7.1.1.4. Change directory

cd

7.1.1.5. Change to parent directory

cd..

7.1.1.6. Show the path where actually you are

pwd

7.1.1.7. Create a directory

mkdir

7.1.1.8. Remove directory

rm -r

7.1.1.9. Force remove

rm -f

7.1.1.10. file

cp

7.1.1.11. Rename or move file

mv

7.1.1.12. Create file

touch file

7.1.1.13. Output contents of file

cat [file]

7.1.1.14. Write standard input into file

cat > [file]

7.1.1.15. Append standard input into file

cat >> [file]

7.1.1.16. Output contents of file it grows

tail -f [file]

7.1.2. Networking

7.1.2.1. Ping host

ping [host]

7.1.2.2. Show the actually routing table

route -n

7.1.2.3. Check your iptable's rules

iptables -L

7.1.2.4. List all ports

netstat -a

7.1.2.5. Got whois for domain

whois [domain]

7.1.2.6. Get DNS for domain

dig [domain]

7.1.2.7. Reserve lookup host

dig -x [host]

7.1.2.8. Download file

wget [file]

7.1.2.9. Recursively download files from url

wget -r [url]

7.1.2.10. Outputs the webpage from url

curl [url]

7.1.2.11. Connect to host as user

ssh user@host

7.1.2.12. Connect using port

ssh -p [port] user@host

7.1.2.13. Connect and use bind port

ssh -D user@host

7.1.3. Processes

7.1.3.1. Display currently active processes

ps

7.1.3.2. Detailed outputs

ps -aux

7.1.3.3. Kill process with process id (pid)

kill [pid]

7.1.3.4. Kill all processes named proc

killall proc

7.1.4. System Info

7.1.4.1. Show current date/time

date

7.1.4.2. Show uptime

uptime

7.1.4.3. Who you're logged in as

whoami

7.1.4.4. Display who is online

w

Display cpu info

cat /proc/cpuinfo

7.1.4.5. Memory info

cat /proc/meminfo

7.1.4.6. Show memory and swap usage

free

7.1.4.7. Show directory space usage

du

7.1.4.8. Displays readable size in GB

du -sh

7.1.4.9. Show disk usage

df

7.1.4.10. Show kernel config

uname -a

7.1.5. Compressing

7.1.5.1. Tar files into file.tar

tar -cf [file.tar] [files]

7.1.5.2. Untar into current directory

tar -xf [file.tar]

7.1.5.3. Show contents of archive

Show contents of archive Options :

7.1.6. Permissions

7.1.6.1. Change permissions of file

chmod [rights] [file]

4 - read(r)

2 - write(w)

1 - execute(x)

order : owner / group / world

chmod 777 - rwx for everyone

chmod 755 - rw for owner, rx for group and world

7.1.7. Others

7.1.7.1. Search in files for pattern

grep '[pattern]' [files]

7.1.7.2. Search for pattern recursively in dir

grep -r '[pattern]' dir

7.1.7.3. Find all instances of file

locate [file]

7.1.7.4. Show possible locations of app

whereis [app]

8. Editors

9. Docker

9.1. Docker for Data Science

Found this image that is a good starting point for data science projects: You need docker allready installed on the machine.

Installation link and instructions can be found here:

https://docs.docker.com/get-docker/

https://hub.docker.com/r/civisanalytics/datascience-python

  1. docker pull civisanalytics/datascience-python
  2. docker run -p 8888:8888 -i -t civisanalytics/datascience-python:latest /bin/bash In the container, you start the jupyter notebook server.
  3. jupyter notebook --ip 0.0.0.0 --no-browser --allow-root

  4. Access the notebook server at http://localhost:8888/ The exact link including the token is shown in the terminal.

    !To add a folder to the notebook server you can use the following command: docker run -p 8888:8888 -i -t -v /Users/elvismunteanu/elvism/DataScience:/elvism civisanalytics/datascience-python:latest /bin/bash The command added to second step is -v <Your folder location>:<Folder name and location from the Docker container to pass it in>

9.2. Docker Cheat Sheet

9.2.1. Create Containers without Starting

docker container create [image] docker create [image] [command] Use same options and arguments for docker run Image is downloaded if it is not present and Filesystem is created Containers can be started later with docker start

9.2.2. Start a Container

docker start [container] docker container start [container] Use -a to attach to the container STDOUT and STDERR Use -i to attach to the container STDIN (standard input)

9.2.3. Inspect a Container

docker inspect [container] Prints out the container's configuration

9.2.4. Container Names

--name [name] Use --name to name the container. If not specified, the container will be named after the image. Default Names have a form: _

9.2.5. Run Interactive command

docker run -it python:3 bash

Use -it or --interactive --tty option to create an interactive container. Can be stopped typing exit

9.2.6. Port Mapping

-p 8080:80 -p <host port>:<container port>

Commands examples:
$ docker run -it -p 8080:80 -p 8443:443 nginx
$ docker run -it -p 8000:80 -p 8443:443 nginx
$ docker run -it -p 8888:8888 jupyter/datascience-notebook

9.2.7. To stop all Docker containers, simply run the following command in your terminal

docker kill $(docker ps -q)

9.2.8. If you don’t just want to stop containers and you’d like to go a step further and remove them, simply run the following command

docker rm $(docker ps -a -q)

9.2.9. To remove all Docker images, run this command

docker rmi $(docker images -q)

9.2.10. Bind Mount Host Folders

-v ${PWD}:/app -v <host_folder(Source Path)>:<container_folder(Mount Point)> Host path must be absolute path On windows must start with drive location (e.g. C:/Users/elvis/elvism/DataScience)

  • add :ro at the end to enable read only access

docker run -t -v ${PWD}:/app python:3 bash

Alternative option:

--mount type=bind,source=${PWD},target=/app It uses the same syntax as the -v option, but more verbose.

9.2.11. List Containers

display a list of running containers docker ps docker container ls

Use -a or -all to display all containers, including stopped ones and paused

9.2.12. Run Container in Background or Detached

-d or --detach

9.2.13. Attack to a Container

docker container attach <container> docker attach <container>

Use Inspect Command to get detailed Metadata information about the image Use History Command to ee how the Image was built

9.2.14. Environment Variables in Container

-e <key>=<value> --env-file <file> - read variables from file in Docker host

9.2.15. Attach Commands in Running Container

docker container exec <container> <command> docker exec <container> <command> docker exec -it <container> bash Use -it for interactive mode Use -w to change working directory Use -e <VAR>=<value> to pass environment variables

9.2.16. Pull a Image

docker pull <image> docker image pull <image> Use :latest at the end to pull the latest version of the image Use -a to pull all tags of the image

9.2.17. Rename the image

docker rename <old_image> <new_image> remove image name with docker rmi <image>

9.2.18. Print Container Logs

docker container logs <container_name> docker logs <container> docker logs -f <container> - print in real time, press CTRL+C to stop/ It stops the command not the container Use -t to print timestamps Use --since to print logs since a specific timeframe

9.2.19. Inspect Image Metadata

docker image inspect <image> docker inspect <image> docker image history <image>

9.3. Create a Docker Image with example

Components of a Docker Image:

Image Build Process:

9.3.1. DockerFile Format

FROM <base_image> INSTRUCTION arguments


 more arguments```
`INSTRUCTION arguments # comment`

9.3.1.1. Filesystem Modification Instructions

9.3.1.2. Metadata Modification Instructions

9.3.2. Docker Build Command

docker build -t <image_name> . the dot represent the current folder containing the Dockerfile

9.3.3. ARG instruction

This is usefull when the base arguments are needed to be changed but we do not wanna change them in the Dockerfile ARG base=python ARG tag=3 FROM $base:$tag -> python:3.9 example: docker build -t myimage:3.9 --build-arg tag=3.9 .

9.3.4. Copy instructions

COPY <from> <to> COPY <from> <from> <from> <destination\> To copy multiple files or folders the last argument must be a directory

9.3.4.1. Copy and file ownership

COPY --chown=<user>:<group> <from> <to> user must exist in Image's /etc/passwd group must exist in Image's /etc/group

example: COPY --chown=elvis:elvis /Users/elvis/elvism/DataScience/ /app/ COPY --chown=1000:55 src dst

9.3.5. RUN instruction

Run can be used to install and to start a command in the container

RUN <command> RUN <shell command line>

  • Default Shell is /bin/sh
  • No background Processes are allowed!
  • Only non-interactive commands are allowed!

RUN examples: RUN pip install -r requirements.txt RUN pip install django==3.2.6 RUN conda install -y pandas RUN python manage.py migrate RUN useradd elvis && chown -R elvis:elvis /app > /tmp/logfile

9.3.6. Volume Mount Points

VOLUME <path> Declares a volume mount point at directory Directory is created if it does not exist Why use VOLUME?

  • To share data between containers
  • Data persistence - they are independent of containers
  • High Performance

List of Volume Mount Points docker volume ls Create a Volume Mount Point docker volume create <name> Remove a Volume Mount Point docker volume rm <name>

9.3.7. Docker Hub Push Image to Registry

docker login docker tag <image> <registry>/<image>:<tag> docker push <registry>/<image>:<tag> Docker name must include Docker Hub Account name or namespace

9.3.7.1. Other Image Registries

Other docker registries can be used to push images to. You can use any registry that supports the Docker Registry API.

10. SQL

10.1. POSGRESQL

10.1.1. Update a table that have a column with a null value with a value

 UPDATE table SET col1 = 0 WHERE col1 IS NULL;

11. Resources