Closed juzcho closed 3 years ago
Adding links for the three datasets we are using so far:
kickstarter_large_dataset folder -> https://www.kaggle.com/kemical/kickstarter-projects
kickstarter_small_dataset folder -> https://www.kaggle.com/socathie/kickstarter-project-statistics
indiegogo csv -> https://www.kaggle.com/quentinmcteer/indiegogo-crowdfunding-data?select=indiegogo.csv
See pull request which imports and cleans the IDs of the csv files #21
Let's focus first on joining together the kickstarter_large_dataset and the indiegogo csv then we can add the kickstarter_small_dataset if we can/need...
Columns for the kickstarter_large_dataset:
Columns for the indiegogo csv:
Columns mappings for the kickstarter_large_dataset to indiegogo:
We can use this info to decide which columns we want to keep from each dataset and then after that clean and rename the columns for each dataset to create similar data frames.
Columns we should keep from the indiegogo csv that we need to calculate or add to the kickstarter_large_dataset:
See #23 that creates two smaller data frames based on the matching columns above. Next steps....
Do we just create a whole new csv file with our clean version?
Do we just create a whole new csv file with our clean version?
We could but I think it would be easier to just use them as dataframes in the program once we have them read in. We could also create a SQL database with the cleaned csv data as each table, that way if we add more data sources we could run them through the cleaner and add them to the DB and access the data using the same SQL queries. (I might be making it more complicated than it needs to be with the SQL though)
O
Do we just create a whole new csv file with our clean version?
We could but I think it would be easier to just use them as dataframes in the program once we have them read in. We could also create a SQL database with the cleaned csv data as each table, that way if we add more data sources we could run them through the cleaner and add them to the DB and access the data using the same SQL queries. (I might be making it more complicated than it needs to be with the SQL though)
Okay if that is the case there wont be a need to put them together as one new csv. At most we just put it in the code which columns to remove then make it a new New Dataframe.
O
Do we just create a whole new csv file with our clean version?
We could but I think it would be easier to just use them as dataframes in the program once we have them read in. We could also create a SQL database with the cleaned csv data as each table, that way if we add more data sources we could run them through the cleaner and add them to the DB and access the data using the same SQL queries. (I might be making it more complicated than it needs to be with the SQL though)
Okay if that is the case there wont be a need to put them together as one new csv. At most we just put it in the code which columns to remove then make it a new New Dataframe.
@Narwhilian Exactly, we could make a db and some functions to clean our code that goes in and out of our db. This is the better way to do it. I think we should do that later as an extra nice to have feature. For now I am just reading and cleaner the CSV files like we did in our assignments.
@juzcho Correct, for now we should do as you said. I think we should not put together as one csv but maybe inside of a db later as @Narwhilian suggested.
See #25 where I fixed the date cols in the indiegogo dataset and reordered the columns for both df so they match to make reading quicker.
Closed as we are using large_kickstarter and indiegogo datasets as above
6, #7, #11, #12, #14 are dependent on this one.