Package providing simple Python access to data in:
Sroka library was checked to work for Python >=3.8, <=3.11.
Install requirements and enable custom githooks:
pip install -r requirements.txt
git config --local core.hooksPath .githooks/
Check style with flake8:
flake8 .
Please target Pull Requests against dev
branch.
pip install sroka
pip install git+ssh://git@github.com/Wikia/sroka
in home folder create ~/.sroka_config
(hidden folder) file where you will store:
config.ini
file based on config.sample.ini
with information to access Qubole, MOAT, Athena, S3 and Rubiconclient_secrets.json
for GA accessad_manager.json
for GAM accesscredentials.json
for Google sheets accessbigquery_credentials.json
for BigQuery accessAlternatively, you may set localization of your files during analysis:
from sroka.config.config import setup_env_variables
from sroka.config.config import setup_client_secret
from sroka.config.config import setup_admanager_config
from sroka.config.config import setup_bigquery_config
from sroka.config.config import setup_google_sheets_credentials
setup_env_variables('/file_path/config.ini')
setup_client_secret('/file_path/client_secrets.json')
setup_admanager_config('/file_path/ad_manager.json')
setup_bigquery_config('/file_path/bigquery_credentials.json')
setup_google_sheets_credentials('/file_path/credentials.json')
You should end up with .json (!) file with credentials
config.ini
file:
In order to authorize in Google Sheets you need to generate credentials in Google Console:
You should end up with credentials.json
file that should be downloaded to ~/.sroka_config
folder.
Go to link and follow up instructions
within Setting up authentication
section. You should end up with bigquery_credentials.json
json file that should be downloaded to ~/.sroka_config
folder.
config.ini
fileSettings
, there you have Query result location
.
The name of location without s3://
and /
is what you need.'us-east-1'
config.ini
file in relevant fieldshost
and port
values in the configuration. If it is accessible through a unix socket, you need to provide the path to this socket instead in the unix_socket
configuration field.user
and password
values in the configuration.database
configuration field.If you see an error like ValueError: unknown locale: UTF-8
Please add to ~/.bash_profile
lines like this:
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
If PyYAML
package is not building correctly, it may be caused by the fact that newer versions of pip won’t uninstall the package because it’s handled by disutils. Please install PyYAML
package first with --ignore-installed
flag.
If numpy gets messed up during sroka installation it is probably caused by multiple versions installed. Please uninstall all using pip uninstall and then reinstall latest one.
If you encounter RefreshError similar to
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n "error": "invalid_grant",\n "error_description": "Bad Request"\n}')
, try removing all files from ~/.cache
directory.
All people that contributed to sroka development before going opensource (including CR and QA):