osd (Open-Source Data)

As a programmer myself, I have noticed when training AI's, I have to scour the internet for a file size that fits my recommended size. But here at osd, you can have all sorts of data, for free anywhere to use. This is the Australian version, and I plan to introduce different countries to this project. osd has everything, from books, to datasets, to wiki pages. (some of which are still coming)

DISCLAIMER: ALL FILES THAT ARE TO BE DOWNLOADED ALL FOLLOW THE AUSTRALIAN COPYRIGHT LAW AND AUSTRALIAN PUBLIC DOMAIN. IF YOU ARE BASED OUTSIDE OF AUSTRALIA, YOU ARE RISKING GRAVE CONSEQUENCES BY DOWNLOADING DATA. PLEASE CHECK THE COPYRIGHT LAWS AND PUBLIC DOMAIN LAWS IN YOUR COUNTRY TO ENSURE YOU ARE UNDER THE LAW. WE ARE NOT AT FAULT FOR PEOPLE DOWNLOADING DATA FROM OUTSIDE AUSTRALIA. HOWEVER, THERE ARE SOME DATA THAT YOU CAN DOWNLOAD, BESIDE EBOOKS AND MORE THAT ARE COPYRIGHTED. PLEASE CONTINUE AT YOUR OWN RISK.

If you are based outside of Australia, you are welcome to download and contribute different data by opening an issue and welcomed to try out some ideas you have in mind

Here are the steps to download and contribute to the project.

1. Prequistries

You will need the following for this project

• Python IDE (for example, Visual Studio Code)
• Python 3.12+
• Git

2. Cloning

You will need to clone the files. Use

git clone https://github.com/Peggun/osd

This gets all of the files and ready to set up all of the packages.

3. Setting Up

When you have opened up the project folder, you should see a requirements.txt file. This contains all of the packages needed for this project to run. To install all of them, run

• Windows

  py -m pip install -r win-requirements.txt

  or

  pip install -r win-requirements.txt

• Linux & MacOS

  python -m pip install -r linux-mac-requirements.txt

  This should install of the packages for the project. Now to get all of the data, you can run the scripts in the downloads folder. The names tell you what they download

Environment Variables

In the files, there is a .env.example file. This is an example of a proper .env file. To modify the .env file, just make a copy of the .env.example folder and rename it to .env, and change the variables as you like. The main website when I host it with the first stable release, will be different and only known to me.

4. Running

Now onto the final step. To run the project, just run

py run.py

run.py contains all of the necessary code to initialise the app and the database.

5. Issues

If you have ran through any issues throughout this process. Please read ISSUES.md

6. Contributing

If you would like to join and contribute to this project, please read CONTRIBUTING.md