MapKnitter Upgrades planning (overall)

[jywarren]

MapKnitter upgrades being considered and planned fall into a few different categories:

1. MapKnitter Exporter (#298 and #296)
2. Rails upgrades + testing
3. MapKnitter UI

We'll break these out in more detail in separate issues, but do a brief overview here. For all planning issues, see:

https://github.com/publiclab/mapknitter/projects/1

Some of the changes will be aimed at infrastructure and sustainability, as we can't make big changes or improvements without, for example, better tests and a more up-to-date Rails stack. Others will be bug fixes and cleanup of the existing system. Finally, some changes will include a massive optimization of the export system backed by Google Cloud (or equivalent serverless backend), and some more serious UI changes to the map creation, login, and image upload workflows, as well as the map display pages.

A rough working diagram of MapKnitter 2.5 can be found here: https://docs.google.com/presentation/d/12NU6V9UFeOWKJrql8AIQZI3mKUWTuakF8zrGpOlgrNg/edit#slide=id.g49650b018b_0_0

Here are the overall projects; be aware that we may not prioritize each of these, and may try to focus on some of the most urgent and impactful for starters:

Stand-alone Cloud Exporter project

See #298 for in-depth discussion!

• [ ] Exporter JS UI to initiate, track progress (in progress bar) and report completion - see #326
• [ ] list of exports run in this area and by whom

Rails upgrades

• [x] Tour the code and evaluate (@jywarren, others)
  • [x] deprecate very old MapKnitter 1 templates, controllers (some done in #313, more needed for old Knitter interface, etc)
  • [ ] compile list of routes/URLs to build functional tests around
  • [ ] vastly expand unit tests #304
  • [ ] move lots into models for unit testing
  • [x] get CI running (Travis)
  • [x] add test coverage reporting #303
  • [ ] Map out JS API endpoints of MapKnitter, write functional tests for them:
    • image upload
    • image positioning (corner coordinates)
    • image display at diff. sizes
    • https://github.com/publiclab/mapknitter/blob/20d887f4f3f5cb5302dc9865efe82106650def94/app/assets/javascripts/mapknitter/Map.js#L378-L438, https://github.com/publiclab/mapknitter/blob/20d887f4f3f5cb5302dc9865efe82106650def94/app/assets/javascripts/mapknitter/Map.js#L39-L139
  • [ ] refactor pagination code?
  • [ ] consolidate and reorganize editor vs. viewer vs. maps listing templates, exporter templates
  • [ ] expand test fixtures
  • [x] attempt to remove all Way and Node code (maybe done in #313)
• [x] Test coverage (#304)
• [x] Rails 3 => 5.1 upgrade #305
• [ ] MapKnitter/Plots2 integration: #302

New Rails dev

• [ ] new 'per image' view
• [ ] spam management planning #252
• [ ] PL.org integration - brainstorm #302
• [ ] image collections/groups rather than "maps"
  • [ ] we have "maps" which are collections of images, but it's a belongs_to relationship, not a many-to-many. So, we need to think about how a Collections model in Rails might enable us to do many-to-many collections! (let's open a new issue for this)
  • [ ] ability to see all images (faded maybe?) for a region and "add to this collection/map" by clicking
• [ ] view of all images for given region in maps (faded/greyed if not part of this map)
• [ ] Ability to choose time bounding box (some kind of layer manager maybe needed) - see #382
• [ ] Ability to toggle viewing your images or all images (uploads by other users)
• [x] Ability to select multiple images for export " a collection" (building on multiple image selection in https://github.com/publiclab/Leaflet.DistortableImage/)
• [ ] install skylight
• [x] install Sentry

Image position versioning, latency, etc

• [ ] Rename Warpable to Image?
• [ ] New model ImagePosition, or ImagePlacement? with corner coordinates, user_id, timestamp
• [ ] (further along) plan/conceptualize image update rejection (based on locking or whatever) and a UI display of edit rejection
• [ ] low-latency plan for real-time multi user image placement (locking/unlocking/transactions)
• [ ] give images a parent column so we can track multiple versions of an image and also use image sequencer to modify them and re-upload...
• [ ] sockets? may need to work upstream in https://github.com/publiclab/Leaflet.DistortableImage/... not sure
• [ ] simultaneous multi-user UI, like Google docs
• [ ] edit history/reversion, accept changes -- how to present UI? "list/history" menu?
• [ ] hard to see which images have been recently - may be uploading over and over (pre-detect this?)

Leaflet.DistortableImage upgrades

Moving most to here: https://github.com/publiclab/Leaflet.DistortableImage/issues/126

• Leaflet.DistortableImage
  • [x] multiple image selection
  • simpler menu API to add new tools (even after initialization)
  • [x] debugging of menu placement, hide/show (list out bugs)
  • [x] full res download from menu (https://github.com/publiclab/Leaflet.DistortableImage/pull/100 but needs a way to pass in full-res original image)
  • 'place with geodata' tool, not sure if this lives in MK or LDI, we have some code for it, need to link to it from here
  • [ ] nested menus or modal for more tools?
• [x] ordering? by a layers tab?
  • [x] ability to set order in Leaflet DistortableImage (use z-index) using function like map.order_by(f()); integrate with MK UI
• [x] toolbar is a mess depending on browser
• [x] toolbar issues when you delete things

User Interface (brainstorm here, needs organizing)

• [x] more clearly frame as "Edit the base map"? workflow to Google Maps?
• [x] great presentation of maps - a listing page featuring nearby/related images, maps, people, info (#327)
  • [x] Featuring related work and groups below, nearby people and prompt to reach out
• [x] A starting page with a menu: kite/balloon/pole/drone/window with drawings and a "start uploading images" button (also #327)
• [x] Login flow improvements (OAuth integration?)
• [ ] UI improvements for Editor
  • [ ] combine image display from sidebar vs. modal, simplify
  • [ ] reorderable image sidebar
• [ ] UI self-testing to ID issues
• [ ] not clear who can edit what; what happens if i open someone's map and edit it (have image locking defaults, and edit history/reversion)
• [ ] image upload is slow (look at FileUpload UI)
• [ ] could we begin placing images even before upload is complete? (would we need to save locally in the browser?)
• [ ] interactive walkthrough? tour? showing by doing

Leaflet Environmental Layers

• improved layers browser
• "add your data"
• MapKnitter images layer, or the MK layer displaying individual images at high enough zoom levels

Big ideas

These ideas will likely be too large for this project but let's think about them and plan for the future.

• “Just start uploading images”
• Starts with map of current location as homepage
• Lives on PublicLab.org, or uses PL login (as now)?
• Based around coords, not map names

[jywarren]

This is pretty big, folks! Just a heads up to @icarito @tech4GT @SidharthBansal @gauravano that this is coming up pretty soon. Interested to hear your input!

[SidharthBansal]

I think we should write project titles in this issue and open up issues for each of the above to discuss. If we will discuss projects in this issue then things will become confusing, overlapping, conflicting and will result in diverge discussions. Let's break the above into individual issues and discuss each project mentioned by Jeff above in its separate issue. Few points which I observe:

• Low number of collaborators at mapknitter as compared to plots2. So we need to shift the diversity from plots2 to this repo
• Listing major ideas related to Mapknitter in this issue
• Creating Milestones for each of them.
• After 15 days GSoC & RGSoC proposals will start coming so we need to decide on the projects a little early so that new folks can start writing their proposals.
• maintainability issues like documentation/outreachy in mapknitter repository. Enhancement of readme, wiki pages, milestone pages, labels etc at this repository(even though it seems to be a small task but for the entire repository it is a big task).

[SidharthBansal]

Other ideas:

• Addition of danger bot
• Addition of codeclimate
• Addition of dependabot

[grvsachdeva]

Hey @jywarren @SidharthBansal,

There are some projects which can be formed from this issue like Rails upgrade, User Interface, etc. I think we should divide this one into more planning issues.

Regarding installation of dangerbot, codeclimate and dependabot I have opened issue Sidharth. Thanks!

[jywarren]

Yes thanks! but also wanted to have one overview issue to talk about the big picture and if this overall plan sounds doable and reasonable and if we're forgetting anything or if there are other major reorganizing ideas we could consider here too. Thanks!!!

On Fri, Jan 25, 2019, 6:02 PM Gaurav Sachdeva <notifications@github.com wrote:

Hey @jywarren https://github.com/jywarren @SidharthBansal https://github.com/SidharthBansal,

There are some projects formed from this issue like Rails upgrade, User Interface, etc. I think we should divide this one into more planning issues.

Regarding installation of dangerbot, codeclimate and dependabot I have opened issue Sidharth. Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter/issues/300#issuecomment-457726834, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJyVbvOp4MSSs0hpLGU4v9Fv315R3ks5vG3C-gaJpZM4aN-or .

[SidharthBansal]

I am trying to shift some portion of the crowd from plots2 to mapknitter also so that this repository will become active as plots2, image-sequencer is. I hope @jywarren, @gauravano @ebarry you will like this.

[jywarren]

Great work.

[jywarren]

Made a project to track some of this work!!!

[jywarren]

Opened a more detailed list here for Leaflet.DistortableImage, the UI for map knitting: https://github.com/publiclab/Leaflet.DistortableImage/issues/126

[jywarren]

@kayrufty wrote in with some great UI notes -- just noting to Kay that we have issues open for some, and not others. I'll note connected issues where they exist, and we can break out new ones as we go. THANKS!!!

• Lock background: On a couple occasions I accidentally moved the google maps background and it was difficult to get it back perfectly aligned with my pictures.

...actually this one, i'm a bit surprised that it can become unlinked with the images. any clues or screenshots on how this happened, or what led up to it so we can try to reproduce it? Thanks!

• Blink images on or off: There is already a feature to make images more transparent, but it might be handy to have it go as far as to disappear except for maybe an outline so you can more clearly see the google maps background. (https://github.com/publiclab/Leaflet.DistortableImage/issues/6)
• Coordinates: It might be useful to have access to the coordinates from google maps to help identify specific locations that are known on images. (adding this to the above list of UI fixes)
• Zoom Control: I was zooming in and out using my scroll wheel and there were times when I couldn't quite get the right magnification and would have liked a physical magnification bar or even an area to input a number %. (this too!)
• I noticed when I exported the images to a .jpg after manipulating them the images were reordered from what I’d done in mapknitter and it brought forward an image I wasn’t able to match well to the landscape. Example of screenshot of the map knitter page and the shifted image file are attached. (Most obvious if you look at the road in the upper right) (ordering -- #116)

[IshaGupta18]

Yes I think this would be very much needed to make MapKnitter as good as plots2. I am interested in tackling and brainstorming on some of these projects,especially the ones related to MapKnitter UI and I also wanted to chime in some suggestions for UI design for it to be of efficient and smooth use to the public, as I think a lot of brainstorming can be done on making the website easy to use and better organised. I will try to open some planning issues regarding the same for us to discuss there. Does this sound okay @jywarren ?

[jywarren]

Hi @IshaGupta18 -- thanks, that would be great! If you'd like to chime in on the Exporter UI discussion here, that'd be awesome: https://github.com/publiclab/mapknitter/issues/326

I think we could also start one called MapKnitter Editor UI discussion where we can discuss the basic editor layout here: https://mapknitter.org/maps/cranston-test (say, UI for the Uploading, Image listing, Exporting, and the main Leaflet map editing areas). We could copy in some of these issues for discussion there:

* [ ]  image collections/groups rather than "maps"  
  * [ ]  ability to see all images (faded maybe?) for a region and "add to this collection/map" by clicking
* [ ]  view of all images for given region in maps (faded/greyed if not part of this map)
* [ ]  Ability to choose time bounding box (some kind of layer manager maybe needed)
* [ ]  Ability to toggle viewing your images or all images (uploads by other users)
* [ ]  Ability to select multiple images for export " a collection"

[jywarren]

Also note that I've built in a few of these projects with more detail on our Summer of Code program Ideas page: https://publiclab.org/gsoc-ideas, including:

• MapKnitter Synchronous Editing
• MapKnitter Cloud Exporter
• MapKnitter UI (a collection of projects)
• MapKnitter Image Management

[Divy123]

@jywarren one thing that I went through the ideas for image-sequencer project. Is that a single idea that is mentioned there under gsoc-ideas or are they multiple?

[jywarren]

Multiple... and as we often get multiple proposals for the same project, we are ok with breaking big ones into pieces. Thanks!

On Thu, Feb 7, 2019, 4:17 PM Slytherin <notifications@github.com wrote:

@jywarren https://github.com/jywarren one thing that I went through the ideas for image-sequencer project. Is that a single idea that is mentioned there under gsoc-ideas or are they multiple?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter/issues/300#issuecomment-461598773, or mute the thread https://github.com/notifications/unsubscribe-auth/AABfJ9GSsfHOhwGXmNo-uYndHTVLNoS8ks5vLJe2gaJpZM4aN-or .

[Divy123]

Thanks .

[IshaGupta18]

@jywarren This would be lovely I think. I will go through all of this is deeper details. Thanks a lot!

[SidharthBansal]

Thanks Isha

On Fri, Feb 8, 2019, 11:59 PM Isha Gupta <notifications@github.com wrote:

@jywarren https://github.com/jywarren This would be lovely I think. I will go through all of this is deeper details. Thanks a lot!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter/issues/300#issuecomment-461899569, or mute the thread https://github.com/notifications/unsubscribe-auth/AUACQztT5KuHY701nngZTR4m7PxwjR_Kks5vLcIJgaJpZM4aN-or .

[jywarren]

Added more info on Collections model which would be related many-to-many to Images, instead of Warpables that belong to only one Map.