Shortlinks job broke

[benhylau]

^{This initial comment is collaborative and open to modification by all.}

Task Summary

🎟️ Re-ticketed from: #
🗣 Loomio: N/A
📅 Due date: N/A
🎯 Success criteria: Fix it.

https://circleci.com/gh/hyphacoop/worker-coop-scripts

Took a brief look, seems like a API connection error with Rebrandly and without much info to debug from.

To Do

[dcwalk]

Moment it failed: https://circleci.com/gh/hyphacoop/worker-coop-scripts/1163 11 days ago (00:42)

Seems like it fails after consistently after "Updated shortlink: working-open"

[YurkoWasHere]

• Logged in to rebrandly
• Checked thresholds, low everything seems find
• confirmed rebrandly key is correct

💭 Dies line # 34 . Last good one is 33 it seems. perhaps API Throttling of some sort?

[benhylau]

I think the next step should be to:

1. Install spreadsheet2shortlinks locally https://github.com/hyphacoop/spreadsheet2shortlinks#installation

2. Run this to see if it fails on same line https://github.com/hyphacoop/worker-coop-scripts/blob/master/.circleci/config.yml#L96-L99

3. Add a delay in this loop https://github.com/hyphacoop/spreadsheet2shortlinks/blob/master/spreadsheet2shortlinks/cli.py#L202 (maybe rebrandly is throttling when may calls are made to it very quickly?)

[YurkoWasHere]

After some failed attempts at installing script locally started poking at the CSV file.

Last reported line entry #33 Test1: Swiched # 32 and 33 - both showed in new order - no change Test2: Swiched # 33 and 34 - Showed one less line (stopped at #32) Test3: Moved #33 to bottom of file - Showed more lines stopped around 42 Test4: Moved offending line to bottom seemed to work untill those line

Found offending lines

calls,https://jitsi.hypha.coop/hyphacoop
passwords,https://pass.hypha.coop

No idea why they are offending.

[YurkoWasHere]

Interesting: both offending urls are on the same subset.

[ASoTNetworks]

I try adding those URLs in rebrandly front end and it let me. @YurkoWasHere suggested it may be a script issue and not rebrandly

[benhylau]

So does it work now if I make new shortlinks, or do we remove the offending lines, or what?

[YurkoWasHere]

If you add items ABOVE the offending lines it will work

WHY the lines are offending i still do not know.

[benhylau]

I will move passwords and calls to bottom and put new links above that for now.

@dcwalk ^ will unblock your other PR for the moment.

@YurkoWasHere suggested it may be a script issue and not rebrandly

It seems now we need to dig into a script if we want to continue using rebrandly :(

I kinda of like the HTML / JS based approach, as it can be statically generated immediately after merge, and unlikely to fail suddenly for odd reasons... but there are these risks

@dcwalk @YurkoWasHere @ASoTNetworks thoughts?

[benhylau]

Confirming that adding new links work, and fail at the last two as expected:

Updated shortlink: reimbursement
Created shortlink: client-agreement
Traceback (most recent call last):
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python3.6/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python3.6/site-packages/urllib3/connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/lib/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

Also flagging that we have probably invested > 10h on this problem collectively, which seems an appropriate time to discuss phasing out spreadsheet2shortlinks + rebrandly.

[YurkoWasHere]

Will migrate to own solution.

[benhylau]

27 April 2020 Infra Meeting

• decided to get rid of rebrandly, and use hypha.coop host to do nginx redirect
• whatever code we use, commit it to the shortlinks repo, later on build it into ansible

[YurkoWasHere]

Script written to replace long rebrandly pipeline

https://github.com/hyphacoop/shortlinks-site

Hosted same place hypha.coop is located

[benhylau]

So every time it fetches the fresh CSV from GitHub but performance seems ok right?

I think we can talk about what needs to be done at All Hands before closing this and just make sure we update Inventory sheet appropriately before closing.

Thanks!

[dcwalk]

Fixed! Closing and a new issue for any lingering styling