URLs and domain names should be linked in post-processing

romain-fontugne commented 1 year ago

Explain the dataset you want to add and how it would contribute to the Internet Yellow Pages. Currently there is no links between URL nodes and DomainName nodes. I don't think we need to add the domain name for all URLs in the database (so I propose to do this in a post-processing script), but at least we should link URLs and domain names that are matching, for example http://www.iij.ad.jp/en/ and iij.ad.jp should have a link between them.

If possible describe how you would like to model the dataset in the Yellow Pages

What is the label/type of the nodes used by this dataset? The two type of nodes are already existing in IYP (URL and DomainName)
What is the label/type of the links for this dataset? I think the PART_OF label would be a good fit here.
If possible give the cypher description of the links that are desired: Here is an example of the links that would be added:
```
(:URL {url:'http://www.iij.ad.jp/en/'})-[:PART_OF]-(:DomainName {name:'iij.ad.jp'})
```

Notes The required code would be very similar to the one we have for linking IP addresses and prefixes. See iyp/post/ip2prefix.py

Yh010 commented 1 year ago

I would like to work on this issue @romain-fontugne

Yh010 commented 1 year ago

in reference to iyp/post/ip2prefix.py line number 4 : When I cloned the project into my local machine and tried to build the project, I got this error:

PS Microsoft.PowerShell.Core\FileSystem::\\wsl.localhost\Ubuntu\home\yashhegde\ihr-iyp\internet-yellow-pages> pip install radix WARNING: Ignoring invalid distribution -ip (c:\python310\lib\site-packages) WARNING: Ignoring invalid distribution -ip (c:\python310\lib\site-packages) ERROR: Could not find a version that satisfies the requirement radix (from versions: none) ERROR: No matching distribution found for radix is the error only with pip or is radix no longer supported?

romain-fontugne commented 1 year ago

Could you fix that problem? radix should work without problem. That said i never tested it on windows

Yh010 commented 1 year ago

@romain-fontugne i tried installing radix as well as py-radix using pip as an attempt to resolve the issue, but with no success...

Yh010 commented 1 year ago

@romain-fontugne the radix issue is resolved ! radix was installed successfully using the usual commands.Don't know what was causing the issue yesterday

InternetHealthReport / internet-yellow-pages

URLs and domain names should be linked in post-processing #32