BukGet is a JSON API into dev.bukkit.org. The idea is to provide some sort of an interface into DBO's contents as Curse doesn't natively have one for us to interact with.
BukGet is a API into dev.bukkit.org that is trying to parse out the relevant details needed to perform package management for Bukkit. We do not have a client that interacts with our API, however instead offer the API to anyone wishing to use it to build their own package management system for Bukkit. We do our best to try to parse out all of the relevant data, including dependencies, CB build numbers, etc. to help aide developers along in getting the information they need.
We have a JSON API that you can interact with in order to pull the information you need. Details on how to interact with the API are explained later on in this document.
Is your plugin in dev.bukkit.org? If it is, then your plugin is already included.
Visit the Projects main website bukget.org
BukGet utilizes MongoDB as the back-end database and is coded using the bottlepy web framework. For scalability, we use Paste as the application server and front-end that with Nginx for URL routing (in-case we need to stand up a dev instance) as well a logging all of the requests. This implementation framework has been able to handle 20M requests a month without showing any significant impact on the hardware (~10% CPU Utilization and ~1-2% I/OWait as of the writing of this document).
Application List
Script List
Services List
Services Layout (By Server)
The Mongo Database has not had any customization that needs to be documented at this time. The Dallas server is the primary database engine, with the Paris server acting as the secondary. This means that all Database writes has to go through the Dallas Mongo instance.
The Python Installation on the server(s) is the default installation that comes with CentOS 6.3 x64. Pip has also been installed on these hosts manually and is being used to help manage the installation of any needed dependencies as well as the installation of the bukget and bukgen packages themselves. These packages are being pulled directly from the git repository and then being updated using the following command in the appropriate directory:
pip install --upgrade ./
This will install/upgrade any dependencies for the package as well as install the latest version of the package itself.
Below is the configuration for the VHost that we use for BukGet. Keep in mind that minor modifications may exist to also listen for the hostname as well as api.bukget.org
server {
listen 80;
server_name api.bukget.org XXX.api.bukget.org;
access_log /var/log/bukget/api-access.log;
error_log /var/log/bukget/api-error.log;
location / {
proxy_pass http://127.0.0.1:9132/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 0;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
}This script is the main script that is being run to keep bukget up to date. This script runs every 6 hours. It can also be run manually as needed (uncommon). The script has a self-imposed 2 second delay for each URL request so not to overwhelm BukkitDev. It's still worth noting that if a generation is run outside of the normal window, to check to make sure nothing else is currently talking to BukkitDev so now to overwhelm the service we're pulling from.
BukGen also has a number of built-in safeties aside from the 2 second delay. These mechanisms have evolved over the life of the generation script and may change:
Options
This is a script to manually update the document to one or many plugins. There may be many reasons for doing this, however this generally relates to the generator script not being able to pull the right data (this is most common with plugins that are bundled in zip files). To run the script, first you will need to create a json file with the corrected plugin definition and place it into the directory /tmp/bukget/fixed_json. You may need to create this directory if it doesn't exist. The filename of the json files are not important, however they must have the .json extension and be only 1 plugin per json file. When ready, just run the command bukkit_manual. The script should import/update the given plugins then remove the .json files as it runs.
This is the bukget api launcher script. This is generally run from the upstart job, however can be manually run when debugging code to see what the errors are.
This script will update the plugins specified as arguments out-of-band of the normal update cycle. This is useful when a plugin either was never updated, or the update never completed for this plugin. These occurrences are rare, however do happen from time to time.
/opt/bukget/devfiles/force_update.py plugin_name_1 plugin_name_2
This script is designed to run nightly from a cronjob. It walks through the previous days nginx log, determines how many calls to each plugin and from each API were performed, then also updates the popularity scoring based on those. It is not recommended to run this script outside the normal schedule unless it fails as it will modify the popularity scoring for all plugins in the system.