URL shortener

[Logic-gate]

Issue Label:

• [ ] Bug
• [ ] New feature
• [ ] Enhancement
• [x] New component

Description: I have created a url shortener at ki.tc. I made it with CLI in mind. Example Usage:

Create a link:

 curl -i -H "Content-Type: application/json" -X POST -d '{"url": "http://google.com"}' http://ki.tc/

Response:

{
  "url_short": {
    "_id": "b24a138a7f1bce6c493bc", 
    "admin_link": "http://ki.tc/url_shortner/b24a138a7f1bce6c493bc", 
    "link": "http://ki.tc/d93da", 
    "time": "Sun, 27 Aug 2017 18:35:16 GMT", 
    "url": "http://google.com"
  }
}

---

Admin Access:

curl -i -H "Content-Type: application/json" -X GET http://ki.tc/url_shortner/b24a138a7f1bce6c493bc

Response:

{
  "_id": "b24a138a7f1bce6c493bc", 
  "url_build": {
    "_id": "b24a138a7f1bce6c493bc", 
    "admin_link": "http://ki.tc/url_shortner/b24a138a7f1bce6c493bc", 
    "link": "http://ki.tc/d93da", 
    "time": "Sun, 27 Aug 2017 18:35:16 GMT", 
    "url": "http://google.com"
  }, 
  "url_id": 194249668
}

Admin access updated(after someone visits the link). Updated with every Access attempt.

{
  "2017-08-27 18:39:19": {
    "165914": {
      "access_id": "ae3c63d61f35", 
      "access_time": "Sun, 27 Aug 2017 18:39:19 GMT", 
      "ip_address": "X.X.X.X", 
      "user_agent": "Mozilla/5.0 (X11; Linux x86_64)"
    }
  }, 
  "_id": "b24a138a7f1bce6c493bc", 
  "url_build": {
    "_id": "b24a138a7f1bce6c493bc", 
    "admin_link": "http://ki.tc/url_shortner/b24a138a7f1bce6c493bc", 
    "link": "http://ki.tc/d93da", 
    "time": "Sun, 27 Aug 2017 18:35:16 GMT", 
    "url": "http://google.com"
  }, 
  "url_id": 194249668
}

---

Delete Link(DELETE METHOD | admin link)

curl -i -H "Content-Type: application/json" -X DELETE http://ki.tc/url_shortner/b24a138a7f1bce6c493bc

Response:

{
  "result": true
}

---

After finishing, I realised that I spelled shortener wrong shortner. Will stick with it. 😢

If its a bug make sure to include this section.

OS and OS version:

• [ ] Mac
• [ ] Linux 32 Bit
• [ ] Linux 64 Bit
• [ ] Windows 32 Bit
• [ ] Windows 64 Bit

OS Version:

[alexanderepstein]

I will look into adding this as a feature to the short tool. What Im thinking is I will take in the link as an argument and first try to find the true link behind the shortened link. If these two links are the same I will assume the user actually wanted to shorten the provided link and then it will make this api request and return the shortened link. Hows that sound?

[Logic-gate]

Hey,

Why not just pass arguments for shortening and expanding.

short -e http://tinyurl.com/jhkjh78 --> to expand
short -s https://codepen.io/khadkamhn/pen/VvxwKL?depth=everything&limit=all&order=popularity&page=10&q=landing&show_forks=false --> to shorten
short --admin http://ki.tc/url_shortner/b24a138a7f1bce6c493bc --> admin access
OR 
short --admin b24a138a7f1bce6c493bc --> admin access by passing _id
short -d http://ki.tc/url_shortner/b24a138a7f1bce6c493bc --> to delete
OR
short -d b24a138a7f1bce6c493bc --> delete by passing _id

Or something to that affect. Because, checking if a link will resolve to its actual destination will not always work. Take youtube https://youtu.be/aStGnWN-i9o this is a shortened link; let's assume I want to shorten it again, if I were to check if it resolves to itself, it will fail. Case in point:

{
  "/short": {
    "destination": "https://www.youtube.com/watch?v=aStGnWN-i9o&feature=youtu.be", 
    "short_url": "https://youtu.be/aStGnWN-i9o"
  }
}

I think this is a better approach, thoughts?

[Logic-gate]

UPDATE

Since I started the issue, I have made significant changes that improve the overall functionality of ki.tc.

Link creation remains the same. However, admin access now includes fingerprinting from fingerprintjs2.

  "_id": "ab51c4e9ffd3901577973", 
  "ba728db72668": {
    "access_time": "Sun, 10 Sep 2017 10:09:01 GMT", 
    "browser": "chrome", 
    "browser_version": 60, 
    "details": [
      [
        "user_agent ", 
        " XX "
      ], 
      [
        " language ", 
        " XX"
      ], 
      [
        " color_depth ", 
        " XX"
      ], 
      [
        " pixel_ratio ", 
        " XX"
      ], 
      [
        " hardware_concurrency ", 
        " XX "
      ], 
      [
        " resolution ", 
        " XX, XX "
      ], 
      [
        " available_resolution ", 
        " XX,XX "
      ], 
      [
        " timezone_offset ", 
        " XX "
      ], 
      [
        " session_storage ", 
        " XX "
      ], 
      [
        " local_storage ", 
        " XX "
      ], 
      [
        " indexed_db ", 
        " XX "
      ], 
      [
        " open_database ", 
        " XX "
      ], 
      [
        " cpu_class ", 
        " XX "
      ], 
      [
        " navigator_platform ", 
        " XX "
      ], 
      [
        " do_not_track ", 
        " XX "
      ], 
      [
        " regular_plugins ", 
        " XX "
      ], 
      [
        " canvas ", 
        " XX "
      ], 
      [
        " webgl ", 
        " XX "
      ], 
      [
        " adblock ", 
        " XX "
      ], 
      [
        " has_lied_languages ", 
        " XX "
      ], 
      [
        " has_lied_resolution ", 
        " XX "
      ], 
      [
        " has_lied_os ", 
        " XX "
      ], 
      [
        " has_lied_browser ", 
        " XX "
      ], 
      [
        " touch_support ", 
        " XX "
      ], 
      [
        " js_fonts ", 
        " XX "
      ]
    ], 
    "fp": "XX", 
    "ip_address": "XX.XX.XX.XX", 
    "ip_info": {
      "as": "XX", 
      "city": "XX", 
      "country": "XX", 
      "countryCode": "XX", 
      "isp": "XX", 
      "lat": XX, 
      "lon": XX, 
      "org": "XX", 
      "query": "XX", 
      "region": "XX", 
      "regionName": "XX", 
      "status": "XX", 
      "timezone": "XX", 
      "zip": ""
    }, 
    "platform": "linux", 
    "user_agent": "XX"
  }, 
  "url_build": {
    "_id": "ab51c4e9ffd3901577973", 
    "admin_link": "http://ki.tc/url_shortner/ab51c4e9ffd3901577973", 
    "link": "http://ki.tc/18a1f", 
    "time": "Sun, 10 Sep 2017 10:08:49 GMT", 
    "url": "http://www.google.com"
  }, 
  "url_id": 16721477
}

[alexanderepstein]

Sorry I havent responded in a little, I have just started school and was busy working on cryptowatch

Good point about using options I did not account for the possibility of a case like that. When this is implemented that is how it will be done.

Can u further explain the use of this admin link? Why would someone want to use this etc

[Logic-gate]

Hey,

So the admin link is used to track or rather log usage. Again, this is a url shortener with logging capabilities.

Note, it relies heavily on JS; I used the same method t.co uses in twitter, where the page loads, then redirects. The initial loading part takes advantage of fingerprintjs2.

[alexanderepstein]

Sorry I haven't really done anything with this I spent a few hours trying to implement and ran into some errors with how the quotations needed to be escaped when the command is used within a script. It may take a while (since school has started up again and my open source work has been harder to keep up with) but I am definitely looking to implement this.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[alexanderepstein]

Sorry not I've done this yet bot I will do this by the weekend

[alexanderepstein]

So i got everything here to work but the issue seems to be coming from your api, whenver I click on a shortened url all that ends up happening is it decodes the url and adds it to after the ki.tc and gives me a page which says the url was not found ex.

short -s google.com
=====================================================================
Original URL:  google.com
Shortened URL: http://ki.tc/c697f
=====================================================================

Clicking on the shortened url it turn the website into http://ki.tc/google.com and then Im greeted with a webpage that says URL Not found!

For now I will add this to a dev branch but it will not be added to mainstream until this works, afterwards I will look into adding the admin link

[Logic-gate]

Hey, I am looking into the issue; I cant reproduce it.

The curl argument for creating a shortened link is

  curl -i -H "Content-Type: application/json" -X POST -d '{"url": "http://google.com"}' http://ki.tc/

it should, and I tested it multiple times, return a functional shortened link

 HTTP/1.1 201 CREATED
Date: Wed, 08 Nov 2017 19:38:34 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 244
Content-Type: application/json

{
  "url_short": {
    "_id": "sadasdasdasd", 
    "admin_link": "http://ki.tc/url_shortner/asdasdasdasdasdasddasd", 
    "link": "http://ki.tc/fb241", 
    "time": "Wed, 08 Nov 2017 19:38:34 GMT", 
    "url": "http://google.com"
  }
}

Try it http://ki.tc/fb241

I think you might have shortend http://ki.tc/google.com, I'll look into it more this evening.

Cheers,

[alexanderepstein]

I have run my script verbosely and the curl call looks like this

curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "google.com"}' http://ki.tc/

The response looks like this

'HTTP/1.1 201 CREATED
Date: Wed, 08 Nov 2017 20:08:46 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 237
Content-Type: application/json

{
  "url_short": {
    "_id": "58ade014577a9190a6da8", 
    "admin_link": "http://ki.tc/url_shortner/58ade014577a9190a6da8", 
    "link": "http://ki.tc/20434", 
    "time": "Wed, 08 Nov 2017 20:08:46 GMT", 
    "url": "google.com"
  }
}'

Going to the link http://ki.tc/20434 brings me to the page http://ki.tc/google.com

The link you provided to check works, not sure why this issue is occurring when I do it from my script

[Logic-gate]

Ahaaaa, figured it out.

you're posting google.com as oppose to http://google.com; i never wrote a handler for a striped url. should be a quick fix from my part. My bad...

[alexanderepstein]

Ah fair enough, let me know when this is good to go and I will make sure it will be ready to go in my script for the next release :smile:

[Logic-gate]

Hey,

Fixed!!

I opted for adding a url validator(Thanks to your indirect bug report).

    curl -i -H "Content-Type: application/json" -X POST -d '{"url": "asdasdasdasd"}' http://ki.tc/
HTTP/1.1 400 BAD REQUEST
Date: Wed, 08 Nov 2017 22:36:04 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 119
Connection: close
Content-Type: application/json

{
  "error": "malformed url: \"asdasdasdasd\". url must start with a scheme specifier(http://, https://, ftp://...)"

and

curl -i -H "Content-Type: application/json" -X POST -d '{"url": "google.com"}' http://ki.tc/
HTTP/1.1 400 BAD REQUEST
Date: Wed, 08 Nov 2017 22:34:33 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 117
Connection: close
Content-Type: application/json

{
  "error": "malformed url: \"google.com\". url must start with a scheme specifier(http://, https://, ftp://...)"
}

I could have added a regex that simply checks if google.com starts with a scheme, but then I ran into the issue of domains.

Anyway, if this quick fix is something you could work with thumbs-up, if not...let me know..

Cheers,

[alexanderepstein]

I implemented shortening the url by checking to see if the user enters http:// or https:// in the beginning and if not I prepend it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[vitalkanev]

And if I add http:// or https://, this will not work

Here is what I did

```bash vitalkanev@vitalkanev-HP-Artful:~$ short -s https://.xyz '===================================================================== Original URL: https:// Shortened URL: =====================================================================' vitalkanev@vitalkanev-HP-Artful:~$ short -s http://example.com '===================================================================== Original URL: http:// Shortened URL: =====================================================================' vitalkanev@vitalkanev-HP-Artful:~$ short -v Version 1.20.0 ```

[alexanderepstein]

@vitalkanev can you run one of those as bash -x short -s http://example.com

[vitalkanev]

HTTPS output

```bash vitalkanev@vitalkanev-HP-Artful:~/clones$ bash -x short -s https://<...>.xyz + currentVersion=1.20.0 + configuredClient= + configuredPython= ++ uname + [[ Linux != \D\a\r\w\i\n ]] ++ uname + [[ Linux != \D\a\r\w\i\n ]] + getConfiguredPython + command -v python2 + command -v python + configuredPython=python + getConfiguredClient + command -v curl + configuredClient=curl + getopts e:s:uvh opt + case "$opt" in + shorten=true + inputURL=https://<...>.xyz + getopts e:s:uvh opt + [[ '' == \t\r\u\e ]] + [[ 2 == 0 ]] + [[ 2 == \1 ]] + [[ '' == \t\r\u\e ]] + [[ true == \t\r\u\e ]] + [[ curl != \c\u\r\l ]] + checkInternet + httpGet github.com + shortenURL https://<...>.xyz ++ echo https://<...>.xyz ++ grep -Eo '^[h]ttp[s]?://' + newURL=https:// + [[ https:// == '' ]] ++ curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "https://"}' http://ki.tc/ + response='HTTP/1.1 400 BAD REQUEST Date: Tue, 05 Dec 2017 12:21:53 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 115 Connection: close Content-Type: application/json { "error": "malformed url: \"https://\". url must start with a scheme specifier(http://, https://, ftp://...)" }' ++ echo HTTP/1.1 400 BAD $'REQUEST\r' Date: Tue, 05 Dec 2017 12:21:53 $'GMT\r' S' Content-Length: $'115\r' Connection: $'close\r' Content-Type: $'application/json\r' $'\r' '{' '"error":' '"malformed' url: '\"https://\".' url must start with a scheme 'specifier(http://,' https://, 'ftp://...)"' '}' ++ tail -n 1 ++ grep -Eo 'http://ki.tc/[0-9a-zA-Z]*' + returnedURL= + printShortenedResults + cat '===================================================================== Original URL: https:// Shortened URL: =====================================================================' ```

HTTP output

```bash vitalkanev@vitalkanev-HP-Artful:~/clones$ bash -x short -s http://example.com + currentVersion=1.20.0 + configuredClient= + configuredPython= ++ uname + [[ Linux != \D\a\r\w\i\n ]] ++ uname + [[ Linux != \D\a\r\w\i\n ]] + getConfiguredPython + command -v python2 + command -v python + configuredPython=python + getConfiguredClient + command -v curl + configuredClient=curl + getopts e:s:uvh opt + case "$opt" in + shorten=true + inputURL=http://example.com + getopts e:s:uvh opt + [[ '' == \t\r\u\e ]] + [[ 2 == 0 ]] + [[ 2 == \1 ]] + [[ '' == \t\r\u\e ]] + [[ true == \t\r\u\e ]] + [[ curl != \c\u\r\l ]] + checkInternet + httpGet github.com + shortenURL http://example.com ++ echo http://example.com ++ grep -Eo '^[h]ttp[s]?://' + newURL=http:// + [[ http:// == '' ]] ++ curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "http://"}' http://ki.tc/ + response='HTTP/1.1 400 BAD REQUEST Date: Tue, 05 Dec 2017 12:26:03 GMT Server: Apache/2.4.18 (Ubuntu) Content-Length: 114 Connection: close Content-Type: application/json { "error": "malformed url: \"http://\". url must start with a scheme specifier(http://, https://, ftp://...)" }' ++ echo HTTP/1.1 400 BAD $'REQUEST\r' Date: Tue, 05 Dec 2017 12:26:03 $'GMT\r' S' Content-Length: $'114\r' Connection: $'close\r' Content-Type: $'application/json\r' $'\r' '{' '"error":' '"malformed' url: '\"http://\".' url must start with a scheme 'specifier(http://,' https://, 'ftp://...)"' '}' ++ tail -n 1 ++ grep -Eo 'http://ki.tc/[0-9a-zA-Z]*' + returnedURL= + printShortenedResults + cat ===================================================================== Original URL: http:// Shortened URL: ===================================================================== ```

[Logic-gate]

Hey,

It seems that for some reason it's passing an empty url

curl -A curl -s -i -H 'Content-Type: application/json' -X POST -d '{"url": "http://"}' http://ki.tc/

if I am not mistaken, short -s example.com should work. I am assuming that the regex only checks if there is no scheme specifier, https://github.com/alexanderepstein/Bash-Snippets/blob/47e91be373e2099f22e2e9a06c3fb0603f70414a/short/short#L150 and if it actually finds one, it assigns only the specifier https://github.com/alexanderepstein/Bash-Snippets/blob/47e91be373e2099f22e2e9a06c3fb0603f70414a/short/short#L151

I am bad at bash, so I cant do it Adding a condition for when there is a specifier should fix the issues.

Also, this whole thing was due to the addition of a validation function from my part on ki.tc. It only accepts correct scheme specifiers.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has been inactive for 30 days.

[alexanderepstein]

I fixed the url shortener, I am having success expanding urls such as tiny urls but I cannot expand urls that were shortened by your api it just returns a destination identical to the source

[Logic-gate]

Hey, the shortening url feature relies heavily on js, in fact it's dependent on it. After all it is a fingerprinting api. The url expansion api is a simple redirect follower; it wont follow redirects from the shortening api because there won't be any redirects unless js is detected.

It's a shortcoming from my part, not adding non-js redirects for the shortening api.

I will amend it however; in the meantime, curl any shortened url from ki.tc, you'll see the js dump.