search

espin086 / NewsWaveMetrics

is a powerful tool for analyzing news sentiment on both national and local stories, allowing users to correlate these stories with their own uploaded metrics, starting with stock market price data. Stay ahead of the curve and make informed decisions with SentimentSync.
MIT License
2 stars 0 forks source link

Implement Feature to Fetch Economic Data from FRED API and Store in SQLite Database #2

Closed espin086 closed 7 months ago

espin086 commented 8 months ago

Summary We require the development of a feature to fetch economic data using the FRED (Federal Reserve Economic Data) API and store this data in a SQLite database. The data will be stored in a new table named fred and should include a range of economic indicators with varying frequencies (daily, weekly, or monthly).

Detailed Description This feature aims to automate the retrieval of economic indicators from the FRED API, focusing on metrics such as Exchange Rates, Treasury yields, the Fed Funds Rate, CPI, GDP, and more. The data collected will support financial analysis, economic research, and modeling within our application.

Requirements:

Data Fields: Fetch the following economic indicators, prioritizing daily data, but falling back to weekly or monthly frequencies if daily data is not available: Exchange Rates 2 Year Treasury 10 Year Treasury Fed Funds Rate CPI (Consumer Price Index) GDP (Gross Domestic Product) Industrial Production Employment Consumer Sentiment PPI (Producer Price Index) Any additional economic LEADING indicators listed on the Wikipedia page for Economic Indicators. Database Schema: Create a new table named fred in our existing SQLite database. The table should have columns for each of the metrics listed above, along with columns for the date and frequency of the data. FRED API Integration: Utilize the FRED API to fetch the required economic data. Ensure to handle API authentication, rate limits, and data normalization. Data Update Frequency: Implement a mechanism to regularly update the database with the latest available data for each indicator, considering their respective frequencies. Error Handling: Implement comprehensive error handling for API connectivity issues, data parsing errors, and database insertion failures. Documentation: Provide detailed documentation on how to configure the FRED API (including obtaining and setting up API keys) and how to operate the data fetching and storage feature. Acceptance Criteria The system can fetch and store the specified economic indicators from the FRED API into the fred table in the SQLite database. Data is updated according to its availability frequency, with fallbacks to less frequent data if daily data is not available. Proper error handling and logging are in place for troubleshooting. Documentation is complete and clear, enabling easy setup and maintenance of the feature.

ZaibyS commented 8 months ago

@espin086 Today, I conducted research on the FRED API and identified the following datasets relevant to the metrics mentioned:

  1. Exchange Rates
  2. Treasury Yields
  3. Fed Funds Rate
  4. CPI (Consumer Price Index)
  5. GDP (Gross Domestic Product)
  6. Industrial Production
  7. Employment
  8. Unemployment Rate
  9. Consumer Sentiment
  10. PPI (Producer Price Index)

Do you find these datasets suitable for use? Additionally, for CPI and Employment metrics, which dataset should we prioritize?

espin086 commented 8 months ago

for CPI please use: Consumer Price Index for All Urban Consumers: All Items in U.S. City Average (Monthly)

for employment please use: https://fred.stlouisfed.org/series/UNRATE/, just use the unemployment rate, that’s easier, so just ignore the other two you have

espin086 commented 8 months ago

In terms of prioritizing, please get the daily metrics first, then the monthly then all others after that

So for example the Exchange rate is daily so pull that in before you pull in unemployment which is monthl

So prioritize the data that comes in more frequently