Open memulder opened 3 years ago
dt-woods
commented
3 years ago @memulder, yeah, the website I posted for the tutorials has changed. If you go here, you can scroll down to see all the available options. The ones I recommend are in the Products section, such as ArcGIS API for Python, ArcPy, and ArcGIS Online.
dt-woods
commented
3 years ago A few things to note: welcome to the world of finding data. It's what we spend most of our time doing, it's terrible and there's no sugar coating that. I do appreciate the success of the week section and I'm glad some things are clicking. In my professional opinion, it's good to have access to both ArcGIS and QGIS as each has their set of strengths and weaknesses.
dt-woods
commented
3 years ago Sounds good. Thanks, Maddy.
Weekly Report 15 - Week of May 3rd, 2021
Independent Work
Projects
This week's challenges
There were many challenges this week as I got into the nitty-gritty of the special works projects. I really struggled with loading in the XY coordinates for the points in SP2, but Evie and Sidonie were able to help me with that. I also struggled with the same problem many people appeared to be having with the projection of SP2 leading to a huge initial fishnet.
Other than that my biggest challenges, especially as I moved into the projected that were more coding-heavy, had to do with file paths and naming variables correctly. I would often get frustrated because I could not find some data or create a file, but the reason was usually that I had messed up the file path and was looking in the wrong place or I had misspelled/capitalized something when I should not have. It was definitely a frustrating process, but I think I am starting to get a handle on working with python.
This week's successes
I was able to work through all of the special works projects that might be on the final so now I have an idea of how to tackle any of them during the final! I was also able to work together with some other people on the projects which was incredibly helpful, relieved a lot of stress, and led to some better connections. Also I learned a lot of very cool excel tricks to make data manipulation and formatting much more quick and easy.
To-do items
Weekly Report 14 - Week of April 26th, 2021
Independent Work
Projects
This week's challenges
I am so close to having figured out special works project 4, but I am having trouble figuring out how to add the data from each specialist to a dataframe. As it is right now, I can create a data frame with the address, score, latitude, and longitude but only for one specialist at a time (so I have like 1000 different dataframes with a different specialist in each, instead of one dataframe with all of the info for all of the specialists.
I am also really struggling with getting the data from the tables in point form in special works project 2. The XY point data tool is having trouble recognizing my lat and lon fields as possibilities for the X and Y coordinates of my points, but I have reached out to others in the class for some help so hopefully I will be able to come through these issues soon.
Finally, I am also struggling with opening the .ige folder in the do-it-yourself section of the geoprocessing notebook so I havent been able to replicate the process yet.
This week's successes
I finished special works project 1 (though I was only able to completely finish it in Arcpro and not in QGIS since I could not get the point to raster tool to work)! I also made a lot of progress on special works projects 2 - 4. And while working on the projects I learned a lot of nifty Excel tricks (like transposing a table) which are sure to come in handy down the line.
To-do items
Weekly Report 13 - Week of April 19th, 2021
Independent Work
Projects
This week's challenges
This week I focused on creating my podcast and figuring out how to work some of the various recording and editing apps was tricky for me ( for some reason I really struggled with getting various sound clips to play one after the other instead of all at the same time) but I figured out a work around where I was able to compile all of the clips into one file - the pauses between clips are sometimes longer than I would like ideally, but overall I am pretty happy with it.
I also began to build my own geodatabase and watching the video associated with the notebook was incredibly helpful for me but it was hard to have to think three steps ahead to try and figure out what I would need to account for in the database.
This week's successes
I finished my podcast! I have never made a podcase before and while simply recording my voice was not hard, figuring out how to add in sound effects gave me some pause (a lot of the audio editing software was a bit over my head), but I figure out a work around and am pretty happy with the final result. The podcast is a bit Lemony Snicket-esque (though I think my voice is too high and not exaggerated enough to really achieve the atmosphere I was going for) so I hope it goes over well and is as funny to others as I found it to be myself, but its already to be shared on the full moon this coming Monday (a date key to the ending of the podcast).
To-do items
Weekly Report 12 - Week of April 12th, 2021
Independent Work
Organizing AGOL (4 hours) Scripting podcast (1 hour) Performance review write-up ( . 5 hours) Special Works Projects (3 hours)
Projects
This week's challenges
Cleaning up my AGOL turned out to be quite the task ( I will forevermore make sure to immediately update my metadata and organize everything accordingly!). I never realized how many things I had kept there that were half-finished or completely irrelevant (for instance, there were a lot of random feature layers from ESRI trainings from years ago). I had to spend a lot of time trying to figure out what everything was and how best to organize it. Creating folders for each class was very helpful but I could not figure out how to edit my content categories - I do not think I have the privileges requires to change my content categories since no edit button appear when I hover over the title and I would very much like to change them so I can get rid of categories like "lab 8" and replace it with something more relevant.
Aside from the challenges associated with organizing my AGOL I am also struggling with the data for special works 1. The website for the data wont load for me and I think I have found it from another source but the process to download the data is quite involved, so I will have to get some help with this.
This week's successes
I cleaned up my AGOL so now I know what everything is and there is nothing irrelevant there anymore! I also began seriously working on special works projects 2 and 3.
To-do items
Weekly Report 11 - Week of April 5th, 2021
Independent Work
*Videos (2 hours)
7 and #8
Projects
This week's challenges
My main challenge this week (which I am still working on is) has been getting the correct data downloaded and accessible. I am having trouble with the data for Special Works one which I was able to find but I am not sure how to implement the software to make the data file accessible. It can be frustrating with the special works projects (and any GIS project in general) to know what I need to do and not actually be able to implement it over small issues like not understanding the data or something funky going on. Also progress on the special works projects has been slow as I have fallen out of practice with my coding skills so I have had to take the time to go back and remind myself of little tricks and steps to take when coding.
This week's successes
I got cracking on the special works projects finally and I am really excited about them because the topics for each project are so interests and they reminded me of why I love and got into GIS in the first place so the passion and excitement are back! Also thanks to the discussion posts on issues with code in the notebooks I was able to figure out one bit of one of the notebooks proved to be a bit tricky for me and hopefully I will be able to use it to figure out this other problem that has been giving me grief for a few weeks ( I am sure that the problem is very very small, but I cant seem to get past it and I am sure I am making it much more difficult than it needs to be.)
To-do items
Weekly Report 10 - Week of March 29th, 2021
Independent Work
Projects
This week's challenges
This week I revisited the geoprocessing with Arcpy Notebook and I am having trouble with the link to download the zip file - I am trying a work around by downloading the file to my computer and then seeing if I can open the zipfile that way to see if that works, so I will keep you updated on my progress - working on the project and actually trying to write code myself has been a good way for me to really get familiar with everything though.
This week's successes
I did not have any big successes this week, I just got started on a few projects (the podcast, organizing AGOL) since I wasnt able to devote as much time to this class this week as I had so much for my other classes, but I will be able to make more progress this coming week!
To-do items
Weekly Report 9 - Week of March 22nd, 2021
Independent Work
Projects
This week we put the finishing touches on our client projects in terms of the videos, the website, and the Storymap. We also met with our client and I recorded my thoughts on the process in the performance accomplishment document. With the completion of this project I will now pivot from focusing on the client meeting and learning new skills/concepts with the tutorials and notebooks to putting all I have learned into practice by diving into the special works projects.
This week's challenges
I began working through the geoprocessing with Arcpy notebook and trying to recreate the process of creating a vector file which turned out to be a bit challenging. This is a notebook I definitely had to slow down with and really take the time to read everything, revisit some concepts/processes, and walk myself through what some of the outputs meant a couple of times. I wasn't able to finish the notebook this week in terms of creating a vector file, but it is at the top of my to-do list for next week since I really want to get this process down and make sure I am understanding why I am doing what I am doing and not just copying code mindlessly.
On another note, while this is not necessarily a challenge it did give me a bit of (self-imposed) grief this week. I began to sort through my AGOL content and, quite frankly, the amount of content in my account was overwhelming - I had forgotten a lot of the earlier projects I had worked on and I was (and apparently still am) terrible about organizing/naming/tagging/including meta data/etc. so cleaning up AGOL is going to be quite a project for me, but this was definitely a good wake up call for keeping everything tidy in the future.
This week's successes
We successfully held our client meeting! I thought the whole presentation went really well - everyone was engaged and participated and we had a great discussion with Ella afterwards. It is nice to be able to check that task off and start to focus on the special works project and I am very proud of our final product. Also, having to record my experience working on the client project in the performance accomplishment report forced me to revisit the document and I was able to add a bit to other sections and was reminded of other sections to focus on in the future.
To-do items
Weekly Report 8 - Week of March 15th, 2021
Independent Work
Projects
This week we wrapped up a lot of the loose ends for our project - mainly video editing, finishing up our storymap, and discussing the methodology of our article - so now we are all set to present to our client next week (we had to postpone our meeting by a week).
This week's challenges
Figuring out how to reproduce the methodology has been really tough because what is described in the paper is incredibly vague so we weren't sure which tools/processes the authors of our paper actually used to produce their final maps. Ryan and I spent a lot of time running various tools/ combination of tools and brainstorming what to do and I also met with the CGA fellows to try and figure out what to do.
This week's successes
We figured out how to reproduce the methodology!! It turns out we had been right about what tools we were using but we needed to turn the streets layer into a polygon before running the rest of the tools - funny how a simple little thing can throw such a big wrench in your plans.
On another note, I learned how to do some very very basic video editing, and even though the videos are very amateurish it felt really incredible to have successfully figured out how to stitch videos together, add in graphics, title slides, and music, and then how to upload the videos to YouTube to host them on our website. Overall, this week, despite the fact that the client meeting got pushed back a bit, I felt really proud of all the work I (and of course my incredible teammates) had done for this project especially since I tried a lot of new things like using web feature services, video editing, and website creation over the course of this project. This is by far the coolest thing I have ever produced and I had no idea I was capable of doing all this - now I am really looking forwards to presenting at the client meeting!
To-do items
Weekly Report 7 - Week of March 8th, 2021
Independent Work
Projects
Next week we have our client meeting so this week we were looking to finish up all the remaining loose ends for our project. John created a StoryMap to present our progress to the client while Ryan and I worked to get as far as we could to recreate the maps in our article. We also compiled our videos and began working on the geoethical section of our progress report based on our paper.
This week's challenges
I face two main challenges this week. Firstly, and probably unsurprisingly as I hark on it so much, we ran into some trouble with re-creating our client paper, specifically in trying to recreate the maps for our client meeting paper; the methodology for the paper was very vague (the authors "applied gis tools") and we struggled to figure out which specific tools we should use to create layers that showed the correct proximity calculation and created the correct output in terms of feature type.
I also ran into some challenges when working through the Geoprocessing with the ArcGIS API Notebook. Conceptually I understand what we are doing, but I am sometimes having trouble actually translating what I want to achieve into code so that it does what I want it to (such as when trying to answer the challenge questions). I think I need to better understand how each variable/dataset we are working with is structured so that I know how to access what I want and what the correct syntax is in terms of calling a method/selecting out data.
This week's successes
We are very very close to being all ready for our client meeting next week! All that is left is a little bit of editing and the layer creation (if we can figure it out). As hard as the client meeting project has been in terms of feasibility of reproducibility, it has forced me to learn new skills and brush up on old ones. For instance, I am now very familiar with the spatial analysis tool box and spatial joins (which I embarrassingly sort of forgot about) after trying to recreate the methodology in the client paper and I learned a bit about video editing as I was compiling our videos - something I had absolutely no previous experience in.
To-do items
Weekly Report 6 - Week of March 1st, 2021
Independent Work
Projects
This week's challenges
This week I mostly focused on preparing all of the data for our client meeting and working on the videos, there were no big new challenges, though figuring out how to record myself in my own zoom room did give me a little grief at first (but it was nothing that I couldn't figure out eventually). Next week we are aiming to tie up our loose ends for the client meeting so I hope to get back to some of the more individual tasks and I am sure I will run into some difficulties as I start to code a bit more.
This week's successes
I found a lot of the data we wanted for the client meeting - the data was not from the same sources listed in the paper, but at least we can try to replicate the methodology in our article !!
To-do items
Weekly Report 5 - Week of February 22nd, 2021
Independent Work
Projects
This week I focused on data collection for the project while Ryan worked on the website and John started the weekly blog. We also began scripting out our videos. The team also discussed changing the geography of the article to use Helsinki data since it has been so hard to access and download the Kuopio data, which came as quite a shock to the team.
This week's challenges
Not to be all doom and gloom, but finding the data for the client project has become the bane of my existence. Not only are the sites with the data hard to navigate and the way the data is organized and store very unclear, but all of the data I have (tried) to download is in Finnish, so even though I understand visually what is in front of me I have no clue what is in the attribute table or what the layers are called unless I use the google translate camera feature on my phone, which is an arduous and inaccurate process. Another frustrating part of this data-finding process have been (once I have been able to navigate to the right service) not being able to download the data that I want that is right in front of me or being redirected to multiple other sites in order to get to the download option. Finally, it turns out that you have to either be a licensed realtor or pay to access the real estate data without which it is impossible to carry out the article's methodology.
New Feature: This week's successes
Happily, all the pain caused by the unproductive data searching bore some fruit! In searching for all of the data for our client project I was forced to use web feature services/ web map services and the likes. I had heard about them previously but never actually used them so it was exciting to have to figure out how to use them, though I definitely need to learn more about them. Also I have been working in QGIS a lot more which is a platform I have wanted to become more familiar with for a long time now.
To-do items
Weekly Report 4 - Week of February 15th, 2021
Independent Work
Projects
We met for the second time this week to discuss our timeline for completing the client meeting and decided to aim for the mid-march to complete the project (which is when we signed up to present to the CGA fellows). We formalized our logo and divided up next steps with John and Ryan focusing on the website (which we now have up and running) and me focusing on data collection. So far so good!
This week's challenges
The main focus of this week for me was trying to collect all of the data for the client project. It has been a bit of a challenge to get all of the correct data since the paper is not too detailed about exactly which datasets were used, there are a lot of datasets to find, and the datasets are scattered across multiple data portals (some of which are in Finnish and Google Translate is struggling a bit with some of the webpages). I underestimated how hard finding these datasets would be since it seemed like the paper was so forthcoming about which datasets it used and where the data came from, but I am finding that the paper was too general in referring to the data to make it easy to find the correct data. On top of all of that I also have to keep in mind the correct time period for the data so that our results are as close to the results of the paper as possible.
On another note, I struggled a bit with the tutorial I did this week in that I was never quite sure what I was doing and it took me a while to understand what was going on. I think I will need to re-visit the tutorials later on when I have a bit of a stronger base in coding. As of right now I am able to copy what the tutorial tells me to and achieve the correct final result, but I am not completely understanding all of the jargon or the processes and I couldn't reproduce any of the code on my own especially given that I am not at all familiar with html.
This week I was really confronted by all of the stuff I don't know. I began to dive more deeply into coding with GIS and all of the new concepts, processes, and syntax were a little overwhelming, but I was also super excited by all of the new possibilities. I'll just need to practice and explore a lot more to feel comfortable and competent with all the new stuff!
To-do items
Weekly Report 3 - Week of February 8th, 2021
Independent Work
Projects
On Monday we had our first GPS consultancy meeting where we discussed which paper to choose, which platform we want to use for our website, and on branding for our consultancy (I designed a few logos for the team to look at). Next steps include beginning to design our website and finding the data we want ( or outlining our methodology if we cant find the data). Most importantly we need to sign up for the actual meeting itself soon.
This week's challenges
This week I began to dip my toe into the more advanced coding so I ran into a few more issues than last week. Firstly, I had an issue with permissions in the files and folder notebook but I got that sorted out - turns out I had missed a code block. The other challenges I ran into this week had to do with the ArcGIS Developers tutorials. I did the Create A Custom Basemap tutorial last week and had no issues understanding it, but as I tried to jump into the more advanced querying tutorials I realized I needed to go back and start with the basics even though some of the querying tutorial seemed more interesting to me since I had no idea what code pen was and had never worked with html before. I think code pen is incredibly niffy and useful so I am excited to use it more, but I still do not really understand the syntax of html and I need to learn more about APIs (I am never sure which API to use when first starting a tutorial or even what the differences between the options are)
To-do items
Weekly Report 2 - February 1st, 2021
Independent Work
Projects
This week I joined a consultancy with Ryan and John and we set up our first meeting to discuss the branding for our consultancy and our paper for this coming Monday. I also spent some time looking for potential papers for the project to bring to the team for review.
This week's challenges
The big challenge for me this week was organization. In order to do the Google Takeout task I had to clean up my desktop quite a bit - which will be good long-term and forced me to re-think some of my data storage habits and hoarding tendencies. I haven't yet gotten into the more complex tasks assigned for this class so no technical issues came up, but I imagine they will pop up soon as I embark on some of the more difficult/ new-to-me tasks this upcoming week.
To-do items
Weekly Report 1 - Week of January 27th, 2021
Independent Work
Projects
No projects started yet.
This week's challenges
Besides accidentally texting in the class group chat when I did not mean to, this week has been off to a good start and not too challenging! There has been a lot of information coming my way with the start of the semester and I think it will take some time to get back into the groove of things (and a second look over the syllabi might be good). I was a little overwhelmed with all the tasks for all of my classes staring me down this semester, but I just have to remember to plan everything out and work at a steady state. When I have a lot to do I can often get going too hot out of the gate, so Ill have to remind myself that sometimes it is better to wait to work on something so that I do not get overwhelmed trying to accomplish too much too soon (with more time (and a good plan) comes more resources and information which allow for more success and less stress!).
To-do items
Generally, for next week I want to find a group for the client project, find a cool GIS tool/method to share with the class, and check off two tasks from the Becoming More Geoethical section and four tasks from the Enchancing Your Skills section. I'd like to get some of the smaller tasks done when there is not too much going on yet with my other classes so that I do not have to worry about them later when the semester really starts to pick up