answerquest commented 6 years ago

25.10.18: Edit:

Link to the Slides shown onscreen at the meetup

Gathering problem statements for a hackathon. Guidelines:

One problem statement per post. If you have multiple, then make separate posts.
Clearly show skillsets required to qualify for participating in the task.
Provide reference links if any at bottom.
Make it small, not big. We want people to actually finish the task and put out the solution within a half-day or whatever the hackathon's action duration is. If the task is looking big, break it down into multiple tasks.
We want to solve real world problems with the task. Hence, provide a realistic use case scenario or better a real life need (mention who needs it)

answerquest commented 6 years ago

Best route mapping for a Delivery Van

A van sets out from an origin lat-long location and has a list of lat-longs to visit and then has to return home.
The program should churn out the quickest route, by road, to reach all the points and return back to the origin point.

Probable function call:

routeFinder( { Origin lat:xx.xx, lon:yy.yy } , [ 
     {id:'A1', lat:xx.xx, lon:yy.yy}, 
    { ...(Destination points)... } 
    ]
);

Arguments passed as json. Destination is array of jsons.
Returns a link to a google directions map, or returns a route shape, with the destination points marked with the id key passed.
Another possible output : A PDF or JPEG map ready for printing with the route drawn and destination ID's labeled.

Specifics

We could use google directions, or any open source routing mechanism : whichever is easier. The google map API requirements won't be too high; this program needs to be run only one or a few times in a day.

Reference:

Real world application : MeraFarmer, a social enterprise doing organic foods delivery service in Pune, is looking for this kind of solution to optimize their electric cart's delivery routes and maximize their area and customer coverage (while not running out of battery!), and also want to send their customers the route map with estimated time of arrival.

Sample Data, of actual locations of a delivery run:

latitude,longitude,name
18.538181,73.784732,Origin
18.530648,73.810429,A
18.577498,73.813334,B
18.562004,73.807477,C
18.560634,73.812726,D
18.562955,73.818957,E
18.551968,73.809506,F
18.554745,73.800882,G
18.556018,73.794711,H
18.558204,73.792850,I
18.562328,73.780350,J
18.571946,73.782667,K
18.577635,73.776756,L
18.570844,73.782323,M
18.583362,73.782861,N
18.593711,73.793340,O
18.596440,73.777531,P

answerquest commented 6 years ago

Live location and trip tracking through phone app

An app like GPSLogger on a phone does the tracking.
The app is configured to send the logged location to a web server / http addr etc every 5 seconds or 10 seconds or so.
The trip so far and current location gets published on a map on a web link.
Metrics like time taken to be published.
Coloring of route based on speed, speed color legend

Skillsets needed

Javascript
Leaflet or Google maps API for mapping. Preferably Leaflet.
Python or other server-side script
Web server endpoint making
Familiarity with a GPS logging app that will be good for the job.

Details discussion

The app could be sending a GET request (or an HTTP request.. AFAIK they're the same) to a URL with the lat, long as URL arguments.
Ex: http://123456.herokuapp.com/API/postLocation?lat=xx.xxx&lon=yy.yyy
An HTTP endpoint (possibly a Tornado web class/function) receives the API call, logs the lat & long and timestamp, returns a success message as callback.
The webpage makes a different GET request, and gets all the trip details so far. It refreshes the data every 10 seconds or so.
Next day trip : when server receives a fresh ping from the app after a long period of time (say 4 hrs) then it starts recording a new trip and will give only the new trip to the webpage.
There is be a history function on the webpage to review past trips.
In python3, TinyDB would be a good database engine for this. It stores in a local .json file.
There can be a time slider to take the trip to different time to see where the position was at a particular time.

answerquest commented 6 years ago

Desktop shortcut creator for pyinstaller-created windows executable version of a python app

Use case scenario: python script : name.py

Standalone windows executable is created using pyinstaller at the path /dist/name/name.exe .
The exe has supporting DLLs etc that help it run, at the same place.
The supporting data files, HTMLs, folder structure etc that the original python program uses is all at .
We don't want to move these into ./dist/name/, and we don't want to move all those DLLs etc up. We want to keep executables and data separate, and run things from where they are, which will also be usefull later on for maintenance, re-building new versions etc.
We can't do --onefile to bundle all the DLLs etc into the .exe for a variety of reasons including incompatibility with win7 & lower and threat of false positive virus claim by antivirus programs.
Placing a shortcut to the .exe file at and keeping the "Star in" field blank in the shortcut's properties, seems to do the job. The exe is able to use its supporting DLLs for getting started, and then the program's data files are accessed at only because "Start in" is blank.
Using relative shortcut path in the "Target" field of the shortcut seems to work: %windir%\system32\cmd.exe /c start dist\GTFSmanager\GTFSmanager.exe
But this works only if the shortcut is placed at . If we place the shortcut in desktop, then there is no relative path that we can use from Desktop.. the shortcut needs the absolute "Target" and "Start in" paths. And this will change from system to system. What we want to do

Aim

Create a batch (.bat) file or something that goes places a shortcut to the application on the desktop.
The shortcut should have "Target" field like : D:\blah\blah\<top>\dist\name\name.exe (depending on where the program folder is)
..and "Start in" field as: D:\blah\blah\<top>

References

https://superuser.com/questions/631243/set-start-in-directory-for-a-shortcut-using-batch-commands

answerquest commented 6 years ago

Local video files management with tagging

Skills needed

Python on Windows
Files and folders handling
Video file handling, conversion (ffmpeg)

Task points

Given a configured relative or absolute path, scan all files and folders under it, and pick files with video extensions.
In a web or GUI interface, recreate the files/folders structure, with thumbnails.
Enable the user to edit filenames
Tag videos, arrange in hierarchical categories
... which is used for: searching and displaying things by these tags / categories. Including wordpress-style searching categories where searching the parent shows up its children too but not the other way around.
Duplicate folders, files finder (design or incorporate existing)
Use video processing to create low-D versions of videos for archiving and bigger versions can then be "sacrificed for the greater good"
On importing, recreate the nested folders structure as a category hierarchy and assign them accordingly to the videos.
Don't bother moving videos from their folders when categories etc change later on. Preserve the categorization in the local db.
Enable selecting multiple videos and exporting to a new folder.

References

Organizing Files https://automatetheboringstuff.com/chapter9/
File and Directory Access https://docs.python.org/3/library/filesys.html
Faster file search with pythonhttps://opensourceforu.com/2016/10/file-search-with-python/
Tinydb: fast local json file database, portable, good for not-enormous-db's http://tinydb.readthedocs.io/en/latest/
Tabulator: JS library to show and manage data in tabular form (includes grouping and view images in a column, moving rows between tables): http://tabulator.info/

Real World Application

Adventures Beyond Barriers, a non-profit that works on inclusion for disabled, needs a solution to manage their burgeoning videos collection.
Potential use by practically any team that needs to manage many videos.

answerquest commented 6 years ago

Map based data collection with variable key-value pairs

Have a location picker
A table or something where key-value pairs can be entered, similar to when ediing on OpenStreetMap.
Presets in a dropdown that will add some fixes key-value pairs and populate keys to be given values by the user.
Data is submitted to a noSQL DB.
A view section on the site shows the data thus collected on a map
Enable filtering of data by key-value pairs, similart to when using Overpass Turbo to extract data from OpenStreetMap.
See https://github.com/datameet-pune/datameet-pune.github.io/issues/8

answerquest commented 6 years ago

Select layers in a multi-layered PDF map and generate high-res image with just the selected layers

See https://github.com/datameet-pune/datameet-pune.github.io/issues/12

answerquest commented 6 years ago

Scrape and visualize bank accounts data from PMJDY site

See https://github.com/datameet-pune/datameet-pune.github.io/issues/10

answerquest commented 6 years ago

GUI tool for unpivoting data (and pivoting)

See https://github.com/datameet-pune/datameet-pune.github.io/issues/13 and https://github.com/datameet-pune/datameet-pune.github.io/wiki/Table-Un-Pivot
User uploads a CSV or excel containing pivoted data.
Specifies the columns to be unpivoted, preserved, what the new value field will be etc. Based on pandas' melt command.
Program gives a downloadable CSV that is unpivoted.
Reverse flow too.
These are ready commands in pandas. So let's make a user-friendly GUI for it.
Can become a site on heroku or an standalone executable.

answerquest commented 6 years ago

Ironing out messy Location data

Problem statement

Point locations data having multiple entries for one location
Non-exact but close-by lat-long values, due to different GPS devices mapping same location.
Differing metadata values in the above.
Need to de-duplicate / cluster multiple entries into one.
See https://github.com/datameet-pune/datameet-pune.github.io/wiki/Project%3A-Bus-stops-database

Tasks

Show data in both map and tabular / similar form.
Use filters at both sides to filter data down to the group we need to iron out.
Enable manual selection for inclusion into the group.
Enable to select one lat-long value and impose it on all the entries.
Similarly for other metadata. But not all together. Allow user to iron out one column at a time.
Use geohashes or similar techniques for clustering. See https://github.com/datameet-pune/datameet-pune.github.io/issues/14.

Real world application

Apply to Pune PMPML Bus stops data released on Pune Open Data Portal.

answerquest commented 6 years ago

Visualizing hierarchical json of Pune's Annual Budget

See https://github.com/datameet-pune/datameet-pune.github.io/wiki/Budget-data-visualizer

pankajp commented 6 years ago

Local video files management with tagging

A starter: https://besjournals.onlinelibrary.wiley.com/doi/abs/10.1111/2041-210X.12892 http://vixen.readthedocs.io/en/latest/overview.html

answerquest commented 6 years ago

Linguistic matching to merge datasets

Tamhini >> matches >> Taminhi
Velhe >> matches >> Vhele
तासगांव >> matches >> Tasgaon

Looking up and merging across datasets by doing linguistic matches that can tolerate spelling differences, and possibly even cross language barriers.

For more details see https://github.com/datameet-pune/datameet-pune.github.io/wiki/Match-the-Following

answerquest commented 6 years ago

Legacy font to Unicode office document converter

Let's have a tool that takes a word/excel/powerpoint file and converts all the ShreeDev, Shivaji etc fonts in it to Unicode. The converters are already made; they just need to be applied while preserving formatting.

See: https://github.com/datameet-pune/datameet-pune.github.io/wiki/Legacy-Devnagri-to-Unicode-font-converter-for-formatted-documents-like-word%2C-spreadsheet%2C-presentation

answerquest commented 6 years ago

Link to the Slides shown onscreen at the meetup

screwgoth commented 6 years ago

If anyone wants to help take it forward, the source code for the Django server of the "Live location and trip tracking through phone app" problem is here: https://github.com/screwgoth/live-location-tracker

datameet-pune / datameet-pune.github.io

Problem statements for a hackathon #15