The Igarapé Institute is an independent think and do tank, dedicated to integrating tasks such as security, climate, and development, whose objective is to propose solutions and partnerships for global problems through research, new technologies, public political influence, and communication. In partnership with BCG GAMMA, both proposed the BCG GAMMA Challenge 2021 - Datathon - Virtual Event, to understand Brazilian public security and act in the prevention of violence.
This project is a study proposal, based in Python
and Jupyter Notebook
, which aims at the related tasks performed by BCG GAMMA Datathon.
Major requirements:
Clone the repository and install the requeriments:
git clone https://github.com/dunfrey/BCG-GAMMA-Challenge-2021 bcg-gamma-challenge
cd bcg-gamma-challenge
Integrate BCG GAMMA datasets file:
GAMMAChallenge21Dados.zip
./data
folder ('./data/1. SIM..', './data/2. SINAN..', ...
)$ ls ./data
'1. SIM (Homicidios)'
'2. SINAN (violencia sexual)'
'3. Atlas de Desenvolvimento Humano'
'4. MUNIC'
'5. PNAD Contínuo'
'6. Assistencia Social'
'7. INEP'
'8. Bolsa Familia'
Integrate extra handled datasets from public sources:
handled.zip
from https://drive.google.com/file/d/10a3kaiy3wak-08dI9W2hY-kOZkBj-i-N/view?usp=sharing./data
folder (./data/handled
)$ ls ./data/handled
cbos.parquet # from MTE
sim_pf_homcidios.parquet
sim_municipio_homicidios.parquet
ibge_municipio_populacao_estimada.parquet
ibge_municipio.parquet
ibge_municipio_censo_basico.parquet
ibge_municipio_pib.parquet
inep_municipio_rendimento_escolar.parquet
rais_caged_municipio_funcionarios.parquet
rfb_municipio_cnaes.parquet
municipios_features_labeled.parquet # target poulation needed for 2_data_understanding (1.2, 1.3).ipynb
Install and activate a virtual environment:
Install python requirements:
pip install -r requirements.txt
Reproduce hypothesis 1 validation (social factors):
jupyter notebook notebooks/2_data_understanding (1.2, 1.3).ipynb
Reproduce hypothesis 2 validation (individual factors):
jupyter notebook notebooks/2_data_understanding (2.1).ipynb
Reproduce target population definition:
jupyter notebook notebooks/1.2_data_target_population.ipynb
You can find the installation documentation for the Jupyter platform, on ReadTheDocs. The documentation for advanced usage of Jupyter notebook can be found here.
For a local installation, make sure you have pip installed and run:
$ pip install notebook
Launch with:
$ jupyter notebook
Base folders:
./src/
: contains the scripts that supports the analysis as pre-processing data, plotting graphics, engineering data, helper functions, etc./docs/
: contains the documentation of the project./notebooks/
: contains jupyter notebook experiments needed to collect, explore, prepare data, train and evaluate data../requirements.txt
: contains python dependencies to reproduce the experiments.Explore the folder ./notebooks/
, which contains:
1.1_data_ingesting
: ingestion of public data, which collects, organizes and prepares common data to be investigated throughout the project;1.2_data_target_population
: prepares the target population of the project. In this project, the target population is young adults between 19 and 25 years old and the level of violence related to them;2_data_understanding (1.2, 1.3)
:
2_data_understanding (2.1)
: