New fields in character/regions for map feature

[tPiffrader]

Hopefully the last issue of group 10 :stuck_out_tongue_winking_eye:

• [x] Add a field with an image link to the character scrapper
• [x] Add a boolean field "hasPath" to the character, indicates that we have path information
• [x] Add city character information to the DB
• [x] Add regions/realms to the DB
• [x] Add a field to regions for the coordinates of our polygons

peace :v:

[boriside]

I start modelling this.

[boriside]

@tPiffrader what does it mean: city character information ?

[boriside]

@tPiffrader a region contains many places like cities for example. Don't you need actually the "coordinates" of those places? How do you threat coordinates, like for example how do you estimate, where a city is relatively to the other cities?

[tPiffrader]

@boriside sry but I don't get what you want to say... Can you explain it a bit further?

[tPiffrader]

@boriside we can talk tomo before or after presentation? :)

[kordianbruck]

@tPiffrader please provide the data structure for the remaining fields in json format similar to issue #80

Regions are already in the database: https://got-api.bruck.me/doc/#api-Regions - let us know if this is what you need

[AlexBeischl]

Hi @kordianbruck,

last two: for our realms/regions we need this structure: "name": String (already in the DB), "color": String, "path": JSON Looks in our data like this: {"name": "The Lands of Always Winter", "color": "white", "path":[[84.99013259851141,-98.69138835608068],[84.99013259851141,-141.98102083028493],[84.78666878252469,-141.38368012244283],[84.72242199393571,-141.31340474504964]...]}

Add city character information to the DB: The best solution would be to create a new table: Character's Locations: "name": String "locations": JSON therefore we need a scrapper, which we described to @theocheslerean & @Adiolis today.

[Legenzoo]

@tPiffrader What exactly do you want by: Add city character information to the DB

I could scrap the characters wiki pages for city mentions. (but be aware that there can be false positives)

[gyachdav]

False positives are welcome at this stage. I suggest that any scraped data i.e. Not verified would be coupled with an "annotated" Boolean so we can always point out that shown data requires verification.

[Legenzoo]

@tPiffrader I still require the answer for my question two days ago ;)

Also, what do you mean with: Add regions/realms to the DB There is already such a collection.

Please refine what you actually need.

[AlexBeischl]

Hi Michael, I have described the format for regions/realms we need for polygons here:

"Hi @kordianbruck,

last two: for our realms/regions we need this structure: "name": String (already in the DB), "color": String, "path": JSON Looks in our data like this: {"name": "The Lands of Always Winter", "color": "white", "path":[[84.99013259851141,-98.69138835608068],[84.99013259851141,-141.98102083028493],[84.78666878252469,-141.38368012244283],[84.72242199393571,-141.31340474504964]...]}"

[gyachdav]

status?

[Legenzoo]

What exactly do you want by: Add city character information to the DB

I could scrap the characters wiki pages for city mentions. (but be aware that there can be false positives)

@AlexBeischl: Do you have an answer to this or is this just @tPiffrader ´s issue. He is unfortunately not answering.

[AlexBeischl]

Hi @Adiolis ,

yes the answer I wrote above is for the request of @tPiffrader . "Add city character information to the DB: The best solution would be to create a new table: Character's Locations: "name": String "locations": JSON therefore we need a scrapper, which we described to @theocheslerean & @Adiolis today."

He isn't answering, because I did it for him, sorry for the confusion.

[Legenzoo]

@AlexBeischl We are somehow talking past each other. I need to know if it is sufficient to just look up for each character the cities that are mentioned in the character´s wiki page. So, there can be false positives.

therefore we need a scrapper, which we described to @theocheslerean & @Adiolis today."

is not helpful to me :stuck_out_tongue_closed_eyes:

[Legenzoo]

@sacdallago

for our realms/regions we need this structure: "name": String (already in the DB), "color": String, "path": JSON Looks in our data like this: {"name": "The Lands of Always Winter", "color": "white", "path":[[84.99013259851141,-98.69138835608068],[84.99013259851141,-141.98102083028493],[84.78666878252469,-141.38368012244283],[84.72242199393571,-141.31340474504964]...]}"

Okay, i can create these fields in the model, but there are no api calls more to fill them by the other groups. Also we can´t scrap these data from the wiki. Am i missing here sth.? :laughing:

[AlexBeischl]

Hi @Adiolis "I need to know if it is sufficient to just look up for each character the cities that are mentioned in the character´s wiki page. So, there can be false positives."

Yes, it is not perfect, but the best we can get right now.

"Okay, i can create these fields in the model, but there are no api calls more to fill them by the other groups. Also we can´t scrap these data from the wiki. Am i missing here sth.?"

We have created the data for this ourselfe during the last days and will upload them hopefully tomorrow ( @tPiffrader ). Can you just create such fields and move our data into the DB after we have uploaded them into our git ( I will inform you when we didi it and send you the direct link) ?

[Legenzoo]

@AlexBeischl Jep, i can do that.

[sacdallago]

@Adiolis create a different collection and reference a character by a relation? Similar to twitter and PLOD. This appears to me to be the most database-oriented way of doing this. Then maybe set up routes to get only the paths and routes to insert new ones and update them? This because an editor is on the way.

@AlexBeischl would that work for you? Also. I would keep every path entry separate, aka, one document int the db = [84.99013259851141,-98.69138835608068]

but then there should be something like a counter added to the path in order to reconstruct the order.

This is, to me, the cleanest solution from an outside, pure-logic based view.

What do you guys say?

[Legenzoo]

@sacdallago @AlexBeischl What is the path field in region about? I see an array of coordinate arrays, but no reference to characters at all.

[sacdallago]

@Adiolis I thought

"name": String (already in the DB),

would be the char or episode name?

And as far as I read the data, the coordinates are sequential and defining the path.

I might be totally wrong, let's wait for @AlexBeischl and @mbandle 's answer to this. But let's rather sleep one day on this and get it done correctly than rush into 500 schema and api changes, shall we? :)

[Legenzoo]

@sacdallago:

"name": String (already in the DB),

I thought, that this is just the name of the region, which is already in the db. So, let´s wait =D

I really need some further explanation to this ;)

[sacdallago]

Man, major brainfart... That might be it :D but yes, anyway, different collection, I'd say

[AlexBeischl]

@Adiolis @sacdallago

"What is the path field in region about? I see an array of coordinate arrays, but no reference to characters at all."

The regions is just our realms (so we display a political map, no need for characters or anything else), therefore we just need the name of the region/realm and an array for the coordinates, which we need for building a polygon.

"Also. I would keep every path entry separate, aka, one document int the db = [84.99013259851141,-98.69138835608068]"

For the locations of characters without paths we just need: "name": String "locations": JSON We need the name of the character and then an array for it's locations, which will be displayed as pins. So there is no need for an order (or a counter).

[Legenzoo]

@AlexBeischl Basically i just need to extent the regions collection: https://got-api.bruck.me/api/regions Probably another field name than path would be less confusing :stuck_out_tongue_winking_eye: Basically you draw with the paths the borders of the region. I don´t see a reason to have for this a new collection...

For the location of characters, i can do a new collection..

[AlexBeischl]

@Adiolis And we need a color field for the regions, sry, i forgot to mention it in the previous message. Yeah, you are right, path is a bit confusing, state of brain isn't the best after creating polygones for realms and paths for characters for far too long... :laughing:

[AlexBeischl]

"I don´t see a reason to have for this a new collection..." Adapting it should work for us :smile:

"For the location of characters, i can do a new collection.." :+1:

[sacdallago]

Ok, ok. Happy everything is settled then :) I was just worried about the coming up editor and the possibility to write the new locations via the API to the db, but if the char location is a separate collection, this might actually work majestically :D Also, compliments: it's the first data which is generated and not scraped (except PLOD and sentiment), so this is a biiiig business advantage right there :D

[Legenzoo]

@kordianbruck Please, run npm run refill --collection=characterLocations @AlexBeischl https://got-api.bruck.me/api/characters/locations will then return the whole collection with characters and their locations.

Which api calls are required? Modifying the data via api should not be allowed because of our policy. Edits should be done on the origin source, which is the awoaif wiki.

[Legenzoo]

I created the following api:

• GetByCharacterName:
  https://got-api.bruck.me/api/characters/locations/:name
• GetByLocation: https://got-api.bruck.me/api/characters/locations/byLocation/:location
• GetBySlug: https://got-api.bruck.me/api/characters/locations/bySlug/:slug

All have the ?strict=true option.

@kordianbruck please update the apidoc.

[Legenzoo]

@AlexBeischl If you have your json for the region borders ready, i can finish this issue :smile:

[AlexBeischl]

Hi @Adiolis , thanks :+1: Here is the link to the data: https://raw.githubusercontent.com/Rostlab/JS16_ProjectC_Group10/develop/data/realms.js

[Legenzoo]

@AlexBeischl Okay, i will fill the db with that :dancer:

[bandle]

@Adiolis I have some questions regarding the api: The 3 Functions you mentioned are POST? Could you send me a token to test them? The characterLocation database is filled by one of your scrapers?

Is there a way to add the paths done by us (JSON File) to the DB yet. Maybe I missed it when reading through the post

[AlexBeischl]

Hi @Adiolis I just found one little bug in the realms data and fixed it, can you please update the DB? (should be the last change in those data)

[gyachdav]

cool this time. but going forward please aggregate the bugfixes and commit them to the db only periodically (once a week is optimal)

[Legenzoo]

@mbandle No, they are GET. @kordianbruck haven´t updated the public server yet... And yes, the locations are scraped by the characters´ wiki pages. (looking for city metionings)

@AlexBeischl: Yeah, no problem.

[bandle]

Thanks, this explains why they aren't working at the moment. I looking forward to test it after its published.

What about the character path data? Is there a table yet. We have similar JSON data like the realms. https://github.com/Rostlab/JS16_ProjectC_Group10/blob/develop/data/paths.js

[Legenzoo]

@mbandle Till now no one requested such a feature. Is this required?

@AlexBeischl Please stick to the names of the wiki. In your json you have for example Iron Islands, but the wiki named it The Iron Island. Looking for substrings in the names is really ugly and can lead to false positives.

I outputed via filler the following regions, that are not scraped from the wiki and thus added to the db: Iron Islands, The Lands of Always Winter, Braavosian Coastland, Hills of Norvos, Dhoyne Lands, Andalos, The Golden Fields, The Disputed Lands, Realms of Jhogwin, Ibben, The Forrest of Qohor, The Footprint, Vaes Dothrak, Kingdoms of Ifeqevron, Eastern Grass Sea, Samyrian hills, Bayasabhad, Qarth, Ghiscar, Lhazar, Western Grass Sea, Sarnor, Parched Fields, Elyria, Painted Mountains, Sea of Sighs, Valyria, Western Waste, Rhoynian Veld, Abandoned Lands, Sar Mell

The scraper from the wiki gets: SUCCESS: North of the Wall SUCCESS: The North SUCCESS: The Riverlands SUCCESS: The Vale SUCCESS: The Iron Islands SUCCESS: The Westerlands SUCCESS: The Reach SUCCESS: The Crownlands SUCCESS: Slaver's Bay SUCCESS: Dothraki Sea SUCCESS: Dorne SUCCESS: Free Cities SUCCESS: The Stormlands SUCCESS: Red Waste SUCCESS: Valyrian peninsula SUCCESS: Naath SUCCESS: Isle of Tears SUCCESS: Other SUCCESS: Basilisk Point

Please check for duplicates =/

[bandle]

Yes we need it like mentioned in the first post:

Add a possibility to put character path to the database (preferring a separate table)

Also this has to be linked with the hasPath attribute

Add a boolean field "hasPath" to the character, indicates that we have path information

Someone has already marked it as done

[Legenzoo]

@mbandle okay. Would be interesting to know, who updated this to status finished... I will uncheck it...

@togiberlin @kordianbruck @theocheslerean @boriside @docjag @alschm Who wants to implement this? I really have no time anymore to also do this.

[bandle]

To describe it a little better what has to be done:

• [ ] Table like the other PLOD or sentiment tables with character paths. (path is JSON data)
• [ ] searchable like @Adiolis did it with the character location API
• [ ] path data is provided by our group in the JSON file https://github.com/Rostlab/JS16_ProjectC_Group10/blob/develop/data/paths.js
• [ ] Connect it to hasPath attribute in character table => Each character with path needs hasPath == true https://github.com/Rostlab/JS16_ProjectC_Group10/issues/147

[Legenzoo]

@mbandle Could you please open a new issue for this? This issue is very confusing and too big. Also i don´t want to be assigned to these "new" issues, too. :stuck_out_tongue_closed_eyes:

[AlexBeischl]

@Adiolis I have adapted the names of the realms, which appear in the wiki and are the same as ours. But we have divided some huger regions into different realms. For these I can't adapt the name, furthermore I think combining them into regions wouldn't make sense. So can you add the realms, which don't appear in the wiki (as region) as well?

[Legenzoo]

@AlexBeischl I see the problem. The scraper is using The Iron Islands, but later it is redirected to Iron Islands. I will need to fix that.

I´ve already implemented (localy), that not existing regions in the wiki, are added ;)

[Legenzoo]

@kordianbruck Please refill the regions collection npm run update --collection=regions

I will open for the Iron Island dup thing a new issue, because the problem is independent of this issue. Until the scraper is not fixed, there will be the dup ;) Quick fix would be to edit the json file to the scrapers region name, but this should be fixed today.

@AlexBeischl If you have updates to the data, please use our data/regions.json file. Then, just command kordian to update the regions ;)

[kordianbruck]

Updated all the collections :tada: