Connect App to Dataset and Complete First Data Plot

Description:

When accessing the React frontend, the chronic disease plot fails to render and displays a "Failed to fetch data" message instead. This issue prevents users from visualizing data from the Chronic Disease Indicators dataset: Chronic Disease Data.

Steps to Reproduce:

Start the Application: docker-compose up --build
Access the Frontend: Navigate to http://localhost:3000/ in your browser.
Observe the Error: The designated plot area shows "Failed to fetch data."

Expected Behavior: The React frontend should fetch and display chronic disease data from the provided JSON endpoint in a visual chart.

Actual Behavior: A "Failed to fetch data" message is displayed instead of the chronic disease plot.

Potential Causes

Backend View Issue: scraper/views.py may not be correctly fetching or processing the JSON data.
API Endpoint Error: The /scraper/api/chronic-disease/ endpoint may be malfunctioning.
Frontend API Call Misconfiguration: Incorrect API URL or proxy settings in the React app.
Data Processing Errors: Backend might not be properly formatting the JSON response.

Suggested Fixes:

Modify scraper/views.py: Ensure the ChronicDiseaseDataAPIView correctly fetches and processes data from the Chronic Disease Indicators endpoint.

Example:

import requests
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework import status

class ChronicDiseaseDataAPIView(APIView):
    def get(self, request):
        # might be just https://data.cdc.gov/api/views/g4ie-h725/rows.json
        url = "https://data.cdc.gov/api/views/g4ie-h725/rows.json?accessType=DOWNLOAD" 
        # alternatively csv can be used: https://data.cdc.gov/api/views/g4ie-h725/rows.csv?accessType=DOWNLOAD
        try:
            response = requests.get(url)
            response.raise_for_status()
            data = response.json()
            processed_data = self.process_data(data)
            return Response(processed_data, status=status.HTTP_200_OK)
        except requests.RequestException as e:
            return Response({'error': str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)

    def process_data(self, data):
        columns = data['meta']['view']['columns']
        data_rows = data['data']
        column_map = {column['name']: idx for idx, column in enumerate(columns)}
        target_fields = ['Indicator', 'YearStart', 'DataValue']  # Update based on actual column names
        field_indices = {field: column_map[field] for field in target_fields if field in column_map}
        processed = []
        for row in data_rows:
            entry = {field: row[idx] for field, idx in field_indices.items()}
            if 'DataValue' in entry:
                try:
                    entry['DataValue'] = float(entry['DataValue'])
                except (ValueError, TypeError):
                    entry['DataValue'] = None
            processed.append(entry)
        return processed

Verify API Endpoint: Test the endpoint http://localhost:8000/scraper/api/chronic-disease/ using Postman or curl to ensure it returns the expected data.

Review Frontend Configuration:

Verify the API call in ChronicDiseaseChart.js points to /scraper/api/chronic-disease/.

Rebuild Frontend Docker Image:

docker-compose build frontend
docker-compose up

Ensure that the column names in the backend process_data method match those in the JSON response.
Consider adding logging in the backend to trace data fetching and processing steps.
Update dependencies to address any existing vulnerabilities.

chowe99 / goink