Welcome to PsyyyPsyPsyyyyyduckk - a multifaceted project designed to streamline data handling and analysis.
This 3-in-1 solution encompasses:
PsyyyPsyPsyyyyyduckk is built for performance and reliability, mirroring the unexpected power of Psyduck from the Pokémon universe. Just as Psyduck releases formidable psychic powers when its headaches peak, this project tackles the "headaches" of data analysis by efficiently processing and interpreting large volumes of information.
It's the perfect tool for anyone looking to burst through data-heavy challenges with speed and precision. Dive in and experience a new wave of data management and analysis!
Gave the guidance to use webhooks to process the raw data.
Developed all the Dockerfiles.
Helped a lot with testing and gave suggestions for security measures and new features such as VictoriaMetrics.
Metrics/ is only for Prometheus/VictoriaMetrics types (plaintext).
Currently I've setup dynamic caching TTL which means you don't need to ask for the API data more then the setup cache timers per hour/day/weekly/month.
Quest also has a parameter called Scanned.
It's current logic works for scans based on normal quest timer (00-01 and 10-11):
You can change the procedures:
logic to add more cases or tighten its timers.
Python3 Libraries:
python3.10 -m pip install -r requirements.txt
sudo apt install redis-server
Use bind to allow only designated ips example:
Disable protected-mode:
Properly set a very strong password for redis, because bruteforce can try 150k passwords per second:
Change the port if you want, default is 6379:
cd config/ && cp config.example.json config.json
"MAX_SIZE_GEOFENCE" If you have say 10 geofences, set to 15. Ensuring you have room for more without having to tweak this value often.
"CACHE_GEOFENCES" Time in seconds that it will cache the Geofences.
"REFRESH_GEOFENCES" Time in seconds that it will refresh the cache for the Geofences, should be the same as the above.
"MAX_TRIES_GEOFENCES" Number of attempts on failure.
"RETRY_DELAY_MULT_GEOFENCES" Number of retries multipled on failure. Example with 5 Max Tries and 2 Retry Delay: 1 sec 2, 2 2, so on.
"LOG_LEVEL" INFO/DEBUG/WARNING/ERROR/CRITICAL/OFF choose one.
"CONSOLE_LOG_LEVEL" INFO/DEBUG/WARNING/ERROR/CRITICAL/OFF choose one.
"LOG_FILE" the path inside the repository you want your logs and their names.
"LOG_MAX_BYTES" the max size, currently set to 1 MegaByte by default.
"MAX_LOG_FILES" how many log files it should create (rotating).
"ALLOW_WEBHOOK_HOST" your golbat IP.
"MAX_QUEUE_SIZE" when to flush the Pokémon data to the database, you could edit this value to be higher to create less stress in the database. Example: 2000.
"MAX_QUEST_QUEUE_SIZE" when to flush the Quest data to the database, you could edit this value to be higher to create less stress in the database. Example: 500.
"MAX_RAID_QUEUE_SIZE" when to flush the Raid data to the database, you could edit this value to be higher to create less stress in the database. Example: 500.
"MAX_INVASION_QUEUE_SIZE" when to flush the Invasion data to the database, you could edit this value to be higher to create less stress in the database. Example: 500.
"WORKERS" is set to 1, which by nature is more then enough to process up to 10 Million Raw Data a day.
"MAX_RETRIES" number of attempts on failure.
"RETRY_DELAY" time between each attempt on failure.
"HEADER_NAME" It's up to you if you want to change the Header Name, but I recommend you do.
"SECRET_KEY" Make sure to set your own proper secret key.
"SECRET_HEADER_KEY" Make sure to set your own proper secret header key.
"WORKERS" is set to 1, only ever touch this if you know what you're doing otherwise it might lead to unexpected consumption of resources.
"IP_RESTRICTION" Highly recommended if you don't know what your doing. Make sure to fill in "ALLOWED_IPS" "ip1, ip2, ip3" as in the example.
"ALLOWED_IPS" Requires IP_RESTRICTION = true, set the IPs where you intend to request the API.
"PATH_RESTRICTION" Restricts access to only the API available paths, fully blocking any other attempts.
Take in mind you should allow for the creation of events/procedures in the Database.
cd SQL/ && python3.10 create_database_schema.py
Run Celery the task executioner:
python3.10 celery_worker.py
Always run celery first to check for Database Migrations.
After applying them, you can now populate your total_pokestops table, which is used for stats in the quests, by doing the following:
Important Note: This is a standalone script that should be run once or whenever you want to update your total PokéStops numbers per area.
cd SQL && cp example_obtain_total_stops.py obtain_total_stops.py
Fill in the details for the databases access.
python3.10 obtain_total_stops.py
Run the start_webhookparser.py:
python3.10 start_webhookparser.py
Run the API:
python3.10 start_api.py
npm install pm2@latest -g
Edit ecosystem.config.js as you prefer and run:
pm2 start ecosystem.config.js
cd Psyduck
sudo cp example.database.Dockerfile database.Dockerfile
sudo cp example.celery.Dockerfile celery.Dockerfile
sudo cp example.webhook.Dockerfile webhook.Dockerfile
sudo cp example.api.Dockerfile api.Dockerfile
sudo cp example.docker-compose.yml docker-compose.yml
sudo docker-compose up
#Psyduck stack
services:
psyduck-schema-setup:
build:
context: .
dockerfile: database.Dockerfile
depends_on:
- psyduck-db
Remove this section from the docker-compose.yml after running it the first time.
sudo docker-compose up -d
Your Geofences cannot overlap each other, so kindly create a project and use it where they do not overlap.
When I made this as far as my knowledge goes, Koji did not properly support multi polygons and neither did Dragonite.