Closed ADPennington closed 6 months ago
@abottoms-coder and @jtwillis92 to sync on this issue and refine further per backlog refinement 11.16.21
@abottoms-coder will sync with @jtwillis92 per backlog refinement 11.30.21
@abottoms-coder to refine further at a later time. Maybe belongs in v3 per @ADPennington. Backlog refinement 12.21
@jtimpe this ticket has info on how ofa pulls data down from legacy system: https://github.com/raft-tech/TANF-app/issues/1178
also note: the db returns data for most recent data submission (not all versions).
OFA staff have a unique user/pass, and are provided with host, port and service identifier to connect to db. See example below:
# set env vars
prod_dbuid = glob.os.environ.get('ap_uid')
prod_dbpw = glob.os.environ.get('ap_pwd')
prod_host = glob.os.environ.get('host')
prod_port= glob.os.environ.get('port')
prod_sid = glob.os.environ.get('sid')
# create data source name
dsn_tns = cx_Oracle.makedsn(prod_host, prod_port, prod_sid)
# establish connection to db
connection = cx_Oracle.connect(prod_dbuid, prod_dbpw, dsn_tns)
-- select all T1 records for FY23 for California
SELECT *
FROM tanf.active_family
WHERE fips_code = 6 and rpt_month_year BETWEEN 202210 AND 202309;
-- select all T6 records for FY23 for tribes SELECT * FROM tanf.aggregate WHERE tribe_code > 0 and rpt_month_year BETWEEN 202210 AND 202309;
Follow-on ticket created for Kibana related features. #2840
Note, CSV export of queries is only supported in AWS ES 7.9. However, Cloud.gov only supports AWS ES 7.4. Thus, we can generate equivalent KQL/Elastic queries, but there will not be an easy way to export them until cloud.gov supports a newer version of the AWS ES service, or we are no longer bound to cloud.gov.
Note, CSV export of queries is only supported in AWS ES 7.9. However, Cloud.gov only supports AWS ES 7.4. Thus, we can generate equivalent KQL/Elastic queries, but there will not be an easy way to export them until cloud.gov supports a newer version of the AWS ES service, or we are no longer bound to cloud.gov.
@elipe17 what are the export options with 7.4 ? how would OFA staff retrieve the data to read into our programming tools?
Note, CSV export of queries is only supported in AWS ES 7.9. However, Cloud.gov only supports AWS ES 7.4. Thus, we can generate equivalent KQL/Elastic queries, but there will not be an easy way to export them until cloud.gov supports a newer version of the AWS ES service, or we are no longer bound to cloud.gov.
@elipe17 what are the export options with 7.4 ? how would OFA staff retrieve the data to read into our programming tools?
@ADPennington We can accomplish this using cURL. I will provide instructions in an MD along with some example queries that have been converted and are verified to work.
Description: As captured in #888, we need a permissions group for OFA's data analysts. group name =
DIGIT Team
.Acceptance Criteria:
/admin/data_files/*
Elastisearchpostgres dbTasks:
Supporting documentation
1178
Open questions
DIGIT Team
members access to postgres data? (in absence of elastic/kibana access)