-
admin_events, lab_events, rc_events
All data is packed in the last field.
ETL error
-
-
# Requirements:
- No self-hosting; must be cloud-based
- Guaranteed to process all inputs, not necessarily in order
## Possible technologies/providers:
- Amazon SQS
- Eventador (Kafka-as-a-Serv…
-
## Context
In the new ETL report we have used data source from the old directory (JSSCP), but we should not use the data_sources from different directory so as a part of this card we have to change th…
-
Something like a DataLink replacement but much more pluggable and that left to the user the core actions of assign fields and transform info, basicaly solve the Extract and Load operations and let the…
-
압축한 tar file을 ETL에 제출할 때, 자꾸 뒤에 `-1`, `-2` 같은 postfix가 붙어서 올라가고 있습니다.
로컬 디바이스에서는 파일 이름이 명확하게 `2019-14500.tar` 로 조회되는데,
![Screenshot 2024-04-28 at 5 56 29 PM](https://github.com/SNU-ARC/2024_spring…
-
# Describe your problem or feature you'd like added
Currently data is dumped and reloaded every day. This is not efficient and requires more time to update the data with every course added to MyLA. …
-
![image](https://user-images.githubusercontent.com/92444727/144174944-9986738d-d29a-4f6a-8f02-8954bd6abf1b.png)
-
![image](https://user-images.githubusercontent.com/92928916/143171388-068220b4-572c-4862-8227-f4905ed31458.png)
-
這個issue將被用做記錄kafka-spark-delta ETL進度,我將大致會將其切成四個部分逐一完成,每個部分我再根據工作量切成小的pr進行完成:
- [ ] 讀取prop file配置相關參數 :
透過 prop file 定義過程使⽤的所有參數,包括資料的路徑,資料樣式,目標 Kafka topic相關資訊等。
- [ ] CSV to Spark :
從CSV將資料匯入spa…