[WIP] SD1782 - cache stale headers

[Cmdv]

Part of https://github.com/slamdata/slamdata/issues/1782

This is a rework of what @cryogenian started https://github.com/slamdata/purescript-quasar/pull/87 and @kRITZCREEK helped me to continue.

Not got the tests working as I'm still missing part and think I might have some of the code wrong anyways so stuck a 🚧 [WIP] 🚧

edit: yeah will defo need help with the tests have a look at the output on Travis if you are viewing this 😸

[safareli]

[1/1 NoInstanceFound] test/src/Test/Main.purs:174:5

  174      run isRight $ QF.readFile Json.Precise testFile1 (Just { offset: 0, limit: 100 })

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  No type class instance was found for

    Data.Show.Show { content :: Array Json

                   , expired :: Boolean

                   }

you are getting this as readFile returns ExpiredContent JArray and run ant's it to be Show-able i.e. have Show instane but as ExpiredContent is type alias for record and records as of now have no show instance tests are not compiling. to fix this you might want to map result of readFile where it fails to some value which is Show-able (String for example)

run isRight
  $ QF.readFile Json.Precise testFile1 (Just { offset: 0, limit: 100 })
  <#> \{content, expired} ->  "content: " <> show content <> " expired: " <> show expired

[Cmdv]

Cool thanks @safareli I'll try that out and think that should help me out with others 😺

[Cmdv]

So this gives me:

No type class instance was found for

     Data.Ord.Ord1 (Const Unit)

 while checking that expression #dict Ord
   has type { compare :: ArbMountType -> ArbMountType -> Ordering
            , "Eq0" :: {}
                       -> { eq :: ArbMountType -> ArbMountType -> Boolean
                          }
            }
 in value declaration ordArbMountType

Then the tests fail as expired is returning false when it should be true ¯\_(ツ)_/¯

[Cmdv]

blocked as unable to figure out above issue and test failure.

[garyb]

Did you try adding spy to the expired parsing stuff? The issue above was due to your dependencies being outdated locally (it doesn't fail that way on Travis).

The tests failing here is either because:

1. The expired parsing is wrong
2. The view isn't being configured with the right max-age
3. Quasar doesn't treat 0-max-age views as always being expired after all

[Cmdv]

I can't add the spy and unable to get the test build failure to go, I've done the usual steps of removing everything re installing it etc...

Travis does come up with a load of warning which it didn't in the previous commit so guessing something in this commit changed things https://github.com/slamdata/purescript-quasar/pull/89/commits/1dc7e04fc18b27e80cfc814c41b8808100226bed

[garyb]

The warning are in the library purescript-quickcheck-laws, unrelated to the Ord instance (as per the message about BoundedEnum :wink: ).

[Cmdv]

ah ok I didn't understand those warning messages, just saw a load of BoundedEnum.

[garyb]

I'm making a patch for that library now, although it's unrelated to this it needs fixing anyway.

What's the problem with adding spy/trace?

[garyb]

Patch release made, I've restarted the build so it should be warning free this time I think. Will still fail though 😜

[cryogenian]

There is no any problem with Ord1 0_o

[garyb]

@cryogenian Yeah I mentioned what was going on with that above:

The issue above was due to your dependencies being outdated locally (it doesn't fail that way on Travis).

purescript-const had a lower bound added when I added the stuff that depended on the Ord1 instance being present: https://github.com/slamdata/purescript-quasar/commit/f3c14a4eef160417ab3e02aab6aa2fc0c3a8a033

[cryogenian]

Ehm... view caching was introduced in 21.x version of quasar, could this be a problem with parsing response?

[garyb]

Could be!

[Cmdv]

thanks @garyb turned out there was stuff in the .pulp-cache 😢

[Cmdv]

I've gotten further but I'm unable to get the tests to run properly against a new version of quasar.

[safareli]

Run quasar:

java -Xmx2g -server -jar quasar/quasar.jar -c test/tmp/quasar/config.json -P quasar/plugins/ -L /slamdata -C slamdata

Create mount with Cache-Control header

$ curl -i "http://localhost:53174/mount/fs/testMount4/?request-headers=%7B%22X-File-Name%22:%22test2%22%7D" \
    -H 'Cache-Control: max-age=1' \
    -X POST \
    -d '{"view":{"connectionUri":"sql2:?q=SELECT%20*%20FROM%20%60/smallZips.json%60"}}'

HTTP/1.1 200 OK
Content-Type: text/plain; charset=UTF-8
Date: Fri, 19 Jan 2018 14:35:40 GMT
Content-Length: 23

added /testMount4/test2

Check mount:

$ curl -i "http://localhost:53174/data/fs/testMount4/test2"
HTTP/1.1 200 OK
Content-Type: application/ldjson; mode="readable"; charset=UTF-8
Date: Fri, 19 Jan 2018 14:42:56 GMT
Transfer-Encoding: chunked

{ "_id": "01001", "city": "AGAWAM", "loc": [ -72.622739, 42.070206 ], "pop": 15338, "state": "MA" }
{ "_id": "01002", "city": "CUSHMAN", "loc": [ -72.51564999999999, 42.377017 ], "pop": 36963, "state": "MA" }
{ "_id": "01005", "city": "BARRE", "loc": [ -72.10835400000001, 42.409698 ], "pop": 4546, "state": "MA" }
{ "_id": "01007", "city": "BELCHERTOWN", "loc": [ -72.41095300000001, 42.275103 ], "pop": 10579, "state": "MA" }
{ "_id": "01008", "city": "BLANDFORD", "loc": [ -72.936114, 42.182949 ], "pop": 1240, "state": "MA" }
{ "_id": "01010", "city": "BRIMFIELD", "loc": [ -72.188455, 42.116543 ], "pop": 3706, "state": "MA" }
{ "_id": "01011", "city": "CHESTER", "loc": [ -72.988761, 42.279421 ], "pop": 1688, "state": "MA" }
{ "_id": "01012", "city": "CHESTERFIELD", "loc": [ -72.833309, 42.38167 ], "pop": 177, "state": "MA" }
{ "_id": "01013", "city": "CHICOPEE", "loc": [ -72.607962, 42.162046 ], "pop": 23396, "state": "MA" }
{ "_id": "01020", "city": "CHICOPEE", "loc": [ -72.576142, 42.176443 ], "pop": 31495, "state": "MA" }
{ "_id": "01022", "city": "WESTOVER AFB", "loc": [ -72.558657, 42.196672 ], "pop": 1764, "state": "MA" }
{ "_id": "01026", "city": "CUMMINGTON", "loc": [ -72.905767, 42.435296 ], "pop": 1484, "state": "MA" }
{ "_id": "01027", "city": "MOUNT TOM", "loc": [ -72.67992099999999, 42.264319 ], "pop": 16864, "state": "MA" }
{ "_id": "01028", "city": "EAST LONGMEADOW", "loc": [ -72.505565, 42.067203 ], "pop": 13367, "state": "MA" }
{ "_id": "01030", "city": "FEEDING HILLS", "loc": [ -72.675077, 42.07182 ], "pop": 11985, "state": "MA" }
{ "_id": "01031", "city": "GILBERTVILLE", "loc": [ -72.19858499999999, 42.332194 ], "pop": 2385, "state": "MA" }
{ "_id": "01032", "city": "GOSHEN", "loc": [ -72.844092, 42.466234 ], "pop": 122, "state": "MA" }
{ "_id": "01033", "city": "GRANBY", "loc": [ -72.52000099999999, 42.255704 ], "pop": 5526, "state": "MA" }
{ "_id": "01034", "city": "TOLLAND", "loc": [ -72.908793, 42.070234 ], "pop": 1652, "state": "MA" }
{ "_id": "01035", "city": "HADLEY", "loc": [ -72.571499, 42.36062 ], "pop": 4231, "state": "MA" }
{ "_id": "01036", "city": "HAMPDEN", "loc": [ -72.43182299999999, 42.064756 ], "pop": 4709, "state": "MA" }
{ "_id": "01038", "city": "HATFIELD", "loc": [ -72.61673500000001, 42.38439 ], "pop": 3184, "state": "MA" }
{ "_id": "01039", "city": "HAYDENVILLE", "loc": [ -72.70317799999999, 42.381799 ], "pop": 1387, "state": "MA" }
{ "_id": "01040", "city": "HOLYOKE", "loc": [ -72.626193, 42.202007 ], "pop": 43704, "state": "MA" }
{ "_id": "01050", "city": "HUNTINGTON", "loc": [ -72.873341, 42.265301 ], "pop": 2084, "state": "MA" }
{ "_id": "01053", "city": "LEEDS", "loc": [ -72.70340299999999, 42.354292 ], "pop": 1350, "state": "MA" }
{ "_id": "01054", "city": "LEVERETT", "loc": [ -72.499334, 42.46823 ], "pop": 1748, "state": "MA" }
{ "_id": "01056", "city": "LUDLOW", "loc": [ -72.471012, 42.172823 ], "pop": 18820, "state": "MA" }
{ "_id": "01057", "city": "MONSON", "loc": [ -72.31963399999999, 42.101017 ], "pop": 8194, "state": "MA" }
{ "_id": "01060", "city": "FLORENCE", "loc": [ -72.654245, 42.324662 ], "pop": 27939, "state": "MA" }
{ "_id": "01068", "city": "OAKHAM", "loc": [ -72.051265, 42.348033 ], "pop": 1503, "state": "MA" }
{ "_id": "01069", "city": "PALMER", "loc": [ -72.328785, 42.176233 ], "pop": 9778, "state": "MA" }
{ "_id": "01070", "city": "PLAINFIELD", "loc": [ -72.918289, 42.514393 ], "pop": 571, "state": "MA" }
{ "_id": "01071", "city": "RUSSELL", "loc": [ -72.840343, 42.147063 ], "pop": 608, "state": "MA" }
{ "_id": "01072", "city": "SHUTESBURY", "loc": [ -72.421342, 42.481968 ], "pop": 1533, "state": "MA" }
{ "_id": "01073", "city": "SOUTHAMPTON", "loc": [ -72.719381, 42.224697 ], "pop": 4478, "state": "MA" }
{ "_id": "01075", "city": "SOUTH HADLEY", "loc": [ -72.581137, 42.237537 ], "pop": 16699, "state": "MA" }
{ "_id": "01077", "city": "SOUTHWICK", "loc": [ -72.770588, 42.051099 ], "pop": 7667, "state": "MA" }
{ "_id": "01080", "city": "THREE RIVERS", "loc": [ -72.362352, 42.181894 ], "pop": 2425, "state": "MA" }
{ "_id": "01081", "city": "WALES", "loc": [ -72.20459200000001, 42.062734 ], "pop": 1732, "state": "MA" }
{ "_id": "01082", "city": "WARE", "loc": [ -72.258285, 42.261831 ], "pop": 9808, "state": "MA" }
{ "_id": "01085", "city": "MONTGOMERY", "loc": [ -72.754318, 42.129484 ], "pop": 40117, "state": "MA" }
{ "_id": "01089", "city": "WEST SPRINGFIELD", "loc": [ -72.641109, 42.115066 ], "pop": 27537, "state": "MA" }
{ "_id": "01092", "city": "WEST WARREN", "loc": [ -72.203639, 42.20734 ], "pop": 4441, "state": "MA" }
{ "_id": "01095", "city": "WILBRAHAM", "loc": [ -72.446415, 42.124506 ], "pop": 12635, "state": "MA" }
{ "_id": "01096", "city": "WILLIAMSBURG", "loc": [ -72.77798900000001, 42.408522 ], "pop": 2295, "state": "MA" }
{ "_id": "01098", "city": "WORTHINGTON", "loc": [ -72.931427, 42.384293 ], "pop": 877, "state": "MA" }
{ "_id": "01103", "city": "SPRINGFIELD", "loc": [ -72.588735, 42.1029 ], "pop": 2323, "state": "MA" }
{ "_id": "01104", "city": "SPRINGFIELD", "loc": [ -72.577769, 42.128848 ], "pop": 22115, "state": "MA" }
{ "_id": "01105", "city": "SPRINGFIELD", "loc": [ -72.578312, 42.099931 ], "pop": 14970, "state": "MA" }
{ "_id": "01106", "city": "LONGMEADOW", "loc": [ -72.5676, 42.050658 ], "pop": 15688, "state": "MA" }
{ "_id": "01107", "city": "SPRINGFIELD", "loc": [ -72.606544, 42.117907 ], "pop": 12739, "state": "MA" }
{ "_id": "01108", "city": "SPRINGFIELD", "loc": [ -72.558432, 42.085314 ], "pop": 25519, "state": "MA" }
{ "_id": "01109", "city": "SPRINGFIELD", "loc": [ -72.554349, 42.114455 ], "pop": 32635, "state": "MA" }
{ "_id": "01118", "city": "SPRINGFIELD", "loc": [ -72.527445, 42.092937 ], "pop": 14618, "state": "MA" }
{ "_id": "01119", "city": "SPRINGFIELD", "loc": [ -72.51211000000001, 42.12473 ], "pop": 13040, "state": "MA" }
{ "_id": "01128", "city": "SPRINGFIELD", "loc": [ -72.48890299999999, 42.094397 ], "pop": 3272, "state": "MA" }
{ "_id": "01129", "city": "SPRINGFIELD", "loc": [ -72.487622, 42.122263 ], "pop": 6831, "state": "MA" }
{ "_id": "01151", "city": "INDIAN ORCHARD", "loc": [ -72.505048, 42.153225 ], "pop": 8702, "state": "MA" }
{ "_id": "01201", "city": "PITTSFIELD", "loc": [ -73.24708800000001, 42.453086 ], "pop": 50655, "state": "MA" }
{ "_id": "01220", "city": "ADAMS", "loc": [ -73.117225, 42.622319 ], "pop": 9901, "state": "MA" }
{ "_id": "01222", "city": "ASHLEY FALLS", "loc": [ -73.320195, 42.059552 ], "pop": 561, "state": "MA" }
{ "_id": "01223", "city": "BECKET", "loc": [ -73.12032499999999, 42.359363 ], "pop": 1070, "state": "MA" }
{ "_id": "01225", "city": "CHESHIRE", "loc": [ -73.15796400000001, 42.561059 ], "pop": 3094, "state": "MA" }
{ "_id": "01226", "city": "DALTON", "loc": [ -73.160259, 42.475046 ], "pop": 7357, "state": "MA" }
{ "_id": "01230", "city": "GREAT BARRINGTON", "loc": [ -73.36065000000001, 42.195922 ], "pop": 10603, "state": "MA" }
{ "_id": "01235", "city": "PERU", "loc": [ -73.092433, 42.434604 ], "pop": 2559, "state": "MA" }
{ "_id": "01236", "city": "HOUSATONIC", "loc": [ -73.374544, 42.265296 ], "pop": 802, "state": "MA" }
{ "_id": "01237", "city": "HANCOCK", "loc": [ -73.24873700000001, 42.541961 ], "pop": 2328, "state": "MA" }
{ "_id": "01238", "city": "LEE", "loc": [ -73.231696, 42.298994 ], "pop": 6916, "state": "MA" }
{ "_id": "01240", "city": "LENOX", "loc": [ -73.271322, 42.364241 ], "pop": 5001, "state": "MA" }
{ "_id": "01243", "city": "MIDDLEFIELD", "loc": [ -73.006226, 42.34795 ], "pop": 384, "state": "MA" }
{ "_id": "01245", "city": "WEST OTIS", "loc": [ -73.213452, 42.187847 ], "pop": 329, "state": "MA" }
{ "_id": "01247", "city": "CLARKSBURG", "loc": [ -73.10999, 42.69865 ], "pop": 19054, "state": "MA" }
{ "_id": "01253", "city": "OTIS", "loc": [ -73.082093, 42.18988 ], "pop": 1060, "state": "MA" }
{ "_id": "01254", "city": "RICHMOND", "loc": [ -73.364457, 42.378398 ], "pop": 1134, "state": "MA" }
{ "_id": "01255", "city": "SANDISFIELD", "loc": [ -73.116285, 42.109429 ], "pop": 651, "state": "MA" }
{ "_id": "01256", "city": "SAVOY", "loc": [ -73.023281, 42.576964 ], "pop": 632, "state": "MA" }
{ "_id": "01257", "city": "SHEFFIELD", "loc": [ -73.361091, 42.100102 ], "pop": 1839, "state": "MA" }
{ "_id": "01258", "city": "SOUTH EGREMONT", "loc": [ -73.456575, 42.101153 ], "pop": 135, "state": "MA" }
{ "_id": "01259", "city": "SOUTHFIELD", "loc": [ -73.26093299999999, 42.078014 ], "pop": 622, "state": "MA" }
{ "_id": "01262", "city": "STOCKBRIDGE", "loc": [ -73.32226300000001, 42.30104 ], "pop": 2200, "state": "MA" }
{ "_id": "01266", "city": "WEST STOCKBRIDGE", "loc": [ -73.38251, 42.334752 ], "pop": 1173, "state": "MA" }
{ "_id": "01267", "city": "WILLIAMSTOWN", "loc": [ -73.20363999999999, 42.708883 ], "pop": 8220, "state": "MA" }
{ "_id": "01270", "city": "WINDSOR", "loc": [ -73.04661, 42.509494 ], "pop": 770, "state": "MA" }
{ "_id": "01301", "city": "LEYDEN", "loc": [ -72.60184700000001, 42.601222 ], "pop": 18968, "state": "MA" }
{ "_id": "01330", "city": "ASHFIELD", "loc": [ -72.810998, 42.523207 ], "pop": 1535, "state": "MA" }
{ "_id": "01331", "city": "NEW SALEM", "loc": [ -72.21464400000001, 42.592065 ], "pop": 14077, "state": "MA" }
{ "_id": "01337", "city": "LEYDEN", "loc": [ -72.563439, 42.683784 ], "pop": 2426, "state": "MA" }
{ "_id": "01338", "city": "BUCKLAND", "loc": [ -72.764124, 42.615174 ], "pop": 16, "state": "MA" }
{ "_id": "01339", "city": "HAWLEY", "loc": [ -72.880162, 42.621802 ], "pop": 1325, "state": "MA" }
{ "_id": "01340", "city": "COLRAIN", "loc": [ -72.726508, 42.67905 ], "pop": 2050, "state": "MA" }
{ "_id": "01341", "city": "CONWAY", "loc": [ -72.702473, 42.513832 ], "pop": 1524, "state": "MA" }
{ "_id": "01342", "city": "DEERFIELD", "loc": [ -72.60723400000001, 42.540636 ], "pop": 1281, "state": "MA" }
{ "_id": "01344", "city": "ERVING", "loc": [ -72.41663800000001, 42.604957 ], "pop": 635, "state": "MA" }
{ "_id": "01346", "city": "HEATH", "loc": [ -72.839101, 42.685347 ], "pop": 174, "state": "MA" }
{ "_id": "01349", "city": "MILLERS FALLS", "loc": [ -72.494626, 42.576206 ], "pop": 1893, "state": "MA" }
{ "_id": "01350", "city": "MONROE", "loc": [ -72.960156, 42.723885 ], "pop": 97, "state": "MA" }
{ "_id": "01351", "city": "MONTAGUE", "loc": [ -72.532837, 42.542864 ], "pop": 1699, "state": "MA" }
{ "_id": "01355", "city": "NEW SALEM", "loc": [ -72.306241, 42.514643 ], "pop": 456, "state": "MA" }

I can't see any expiration related headers in response.

[garyb]

Yeah, looks like we need to open a Quasar ticket then!