search

jeff-zucker / solid-file-client

A Javascript library for creating and managing files and folders in Solid data stores
MIT License
55 stars 18 forks source link

v1.0.0 Testing #52

Closed Otto-AA closed 4 years ago

Otto-AA commented 5 years ago

An issue to discuss testing and keep track of testing for the v1.0.0 release

TODOs:

If you think that something is done, just add a comment. If something is missing, just add it.

Otto-AA commented 5 years ago

On mock, I may have a different solution (in the long run) - if solid-rest and its descendants respond to login requests with mock data, we can test against logging in to file:// or app:// and that way, only the base URL needs to be changed to run the test as real against a pod or mock against solid-rest mini-pod.

I also intended to use solid-rest for testing as soon as it is available, that was one of the reasons I advocated for splitting up solid-rest-browser. But I don't think that it would solve the mock issue for account methods. Afaik it doesn't provide popupLogin methods and similar, so we would still need to mock them...?

jeff-zucker commented 5 years ago

Well the tests are using solid-rest-file which is just the un-broken-apart version of solid-rest + solid-file-storage.

Right, there is no popupLogin method currently but I could easily add it to solid-rest.

bourgeoa commented 5 years ago

Completely newbie to test. Can you tell me what I shall read/try to understand .(..jest+...Jekyl)

jeff-zucker commented 5 years ago

[edit:problem mostly solved, see next message but read this one first]

@Otto-AA HELP! I am lost in the difference between node and jest again. I npm'd solid-rest and solid-rest-file and a version of solid-auth-cli which calls them,. All tests for the whole chain pass. But when I have our v1.0.0 of solid-file-client use this new set of packages with jest, I get wierd errors I can't track down. Same tests with a nodejs test script work fine. You should be able to get the new chain of packages by upping the peer and dev dependencies for solid-auth-cli to ^1.0.0 though you may need to remove the old one first.

jeff-zucker commented 5 years ago

OK ,emergency over. Unfortunately the one of only two things which don't work with the new solid-rest is the recursive delete in setup so it was crashing before anything else could run. If I do rm -rf test-folder/SolidApi; npm test ... I got one fail out of 74 tests which is on the recursive put. So other than recursive put and recursive delete, everything seems to work. I don't know if the problem with those is in solid-rest or solid-rest-file or in one of your packages. But with solid-rest up and running, I can add OPTION and mock login/session. Also we can us app:// URLs for our tests if we don't want to use the file - system and run everything in memory as if the files were on disk.

Otto-AA commented 5 years ago

Ok, I will take a look at it. Also if I could change the setup to make it obvious which method is failing.

Otto-AA commented 5 years ago

The problem is, that the new solid-auth-cli responds with a different turtle when reading the /test-folder/

Old turtle:

 @prefix ldp: <http://www.w3.org/ns/ldp#>.
    <> a ldp:BasicContainer, ldp:Container; ldp:contains
    </home/aa/coding/solid/solid-file-client/test-folder/SolidApi>.
    </home/aa/coding/solid/solid-file-client/test-folder/SolidApi> a ldp:BasicContainer; a ldp:Container.

New turtle (solid-auth-cli v1.0.0):

 @prefix ldp: <http://www.w3.org/ns/ldp#>.
    <> a ldp:BasicContainer, ldp:Container; ldp:contains
      <SolidApi>.

Because of this the getFileType and getFolderItems methods also differ and this results in the error in setup.js: Old version of solid-auth-cli:

    folders [ { type: 'folder',
        modified: undefined,
        size: undefined,
        mtime: undefined,
        label: 'SolidApi',
        name: 'SolidApi',
        url: 'file:/home/aa/coding/solid/solid-file-client/test-folder/SolidApi' } ]
    files []

New version:

    folders []
    files [ { type: 'unknown',
        modified: undefined,
        size: undefined,
        mtime: undefined,
        label: 'SolidApi',
        url: 'file:///home/aa/coding/solid/solid-file-client/test-folder/SolidApi',
        name: 'SolidApi' } ]

I haven't taken a look at what the cause for this could be in solid-auth-cli/solid-rest

Otto-AA commented 5 years ago

So basically the readFolder method fails. I'll rewrite setup.js to work even if SolidApi doesn't work. I guess it's stupid to use the class we want to test in the setup of the tests. Will go back to using rimraf for that.

Otto-AA commented 5 years ago

I've updated the original comment to include unsupported tests by solid-auth-cli. They are currently outcommented in the SolidApi.test.js files. As soon as solid-rest supports them we should reenable them.

Otto-AA commented 5 years ago

@bourgeoa We only use jest as a testing library. Here is the getting started doc of jest: https://jestjs.io/docs/en/getting-started.html

The aim of the tests is to automatically find errors in the code. I'll try to explain it shortly with a few steps: First of all we need to define what the library should do, similar to a spec. If we don't know what it should do we can't test for that. For instance: post should resolve with the status 201 when successfully creating a new folder. creating a new folder is done by passing parameters xyz.

Then we need to add tests which try to match the spec. For that we use solid-auth-cli as a client (because it doesn't require the browser) and create a new SolidFileClient/SolidApi instance. Now we try if this instance fulfills the spec by writing several tests. For instance to test that post resolves with 201 when creating a new folder, I call the post method and check if it resolves with 201: test('post resolves with 201 creating a new folder', () => resolvesWithStatus(api.post(dataFolder, getPostOptions(folderName)), 201))

The test(description, function) method is used for testing by jest. The test function will be called and checked whether it throws an error or doesn't. To throw errors it is useful to use the expect function provided by jest (see the getting started doc). For asynchronous code like our api methods it is useful to use the expect(promise).resolves.toBe(expectedValue method of jest (see here). Just make sure that you return a promise if you do this.

In my SolidApi.test.js tests I've added a resolvesWithStatus(promise, status) function to make common tasks easier:

function resolvesWithStatus(promise, status) {
  return expect(promise).resolves.toHaveProperty('status', status)
}

It accepts a promise (e.g. fileClient.get(url)) and checks whether or not it resolves with the status. expect(promise).resolves checks whether the promise resolves or rejects. If it rejects, the test will fail. After .resolves you can add tests from the expect API like toBe(1234) or in our case toHaveProperty('status', 123). This checks if the resolved value has a property called status with the value 123. In a similar manner most tests are done.

jeff-zucker commented 5 years ago

The problem is, that the new solid-auth-cli responds with a different turtle when reading the /test-folder/

I'm pretty sure I've fixed that in latest versions. With solid-auth-cli v1.0.1 you should be good to go. Please let me know if this doesn't fix it.

jeff-zucker commented 5 years ago

OK, going back to basics is helping. I found that POST container when container already exists was broken in solid-rest-file because mkdir errors if you try to create a pre-existing folder it was returning an error. So fixed that by checking for folder existence and only creating if not found . ...

Otto-AA commented 5 years ago

@jeff-zucker Is there already a npm version update for this fix?

Otto-AA commented 5 years ago

So fixed that by checking for folder existence and only creating if not found . ...

The docs for fs recommend to catch EEXIST errors instead of using fs.exists like that:

fs.open('myfile', 'wx', (err, fd) => {
  if (err) {
    if (err.code === 'EEXIST') {
      console.error('myfile already exists');
      return;
    }

    throw err;
  }

  writeMyData(fd);
});

fs.exists is deprecated and listening for errors is probably even more efficient (doesn't need to check in the first hand, but just tries it)

jeff-zucker commented 5 years ago

I am close, but needed to get that two-way testing to make sure. I'll commit that in my morning (about ten hours from now).

jeff-zucker commented 5 years ago

Ha, I first checked for EEXIST but then figured it would be less portable and switched, should have read up first I guess.

jeff-zucker commented 5 years ago

deleteFolderRecursively works too well :smile: I managed to delete my entire local clone of the project. Dropbox saved my butt though.

Otto-AA commented 5 years ago

Yeah... that's why I'd like to "sandbox" the tests into an in-memory pod environment as default. We could also add a safeUrl method for testing which checks if the url is inside of the test-folder folder (and hence prevents accidentally passing /root/xyz as parameter for the deleteFolderRecursively test). But sandboxing as default and then testing in fs/NSS should also be safe enough.

jeff-zucker commented 5 years ago

My test that runs against https:// file:// and app:// now passes all tests so we can def switch to default in-memory test as soon as I npm it all later today.

May I also suggest that we have recursiveDelete default to confirm=true. And if confirm===true, the user is prompted to continue or abort, showing them what they are going to delete. I have a confirm method that works in either browser or command-line if we go this way. Of course that wouldn't work with automated testing so your safe-check would be needed. If we're going to disallow "/" for recursiveDelete anyway, please implement that soon :-)

Otto-AA commented 5 years ago

Regarding the tests/all file from the new commit: It is only testing if solid-auth-cli works properly, nothing specific to SolidFileClient, right? If yes, what would you say about moving it to the solid-auth-cli for testing?

Otto-AA commented 5 years ago

If you intend to extend it to also test SolidFileClient stuff, I'd suggest to create a new testFetch method in tests/utils/ which could be used by all tests (SolidApi.test.js, ...). It could switch between app://... and https requests by changing an environment variable (so we could add scripts for all three in package.json) and also catch requests which would be above the test-folder level. This would also make it easy to rewrite the FolderGenerator I've added to work with all those testing environments.

jeff-zucker commented 5 years ago

It is in the solid-auth-cli tests. I put it there so a) if we run into other issues downstream from solid-file-client, we can quickly add a test and make sure the fix works in all three environments with simple REST commands, then that it works as it shoudld with solid-file-client b) you may spot some things I forgot to test for.

On Fri, Jun 14, 2019 at 10:28 PM A_A notifications@github.com wrote:

Regarding the tests/all file from the new commit: It is only testing if solid-auth-cli works properly, nothing specific to SolidFileClient, right? If yes, what would you say about moving it to the solid-auth-cli for testing?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jeff-zucker/solid-file-client/issues/52?email_source=notifications&email_token=AKVJCJCQQKDDTZZVUYDPSNLP2R4ZFA5CNFSM4HXOKHI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYQZJA#issuecomment-502336676, or mute the thread https://github.com/notifications/unsubscribe-auth/AKVJCJBMBURCT3ZOQI5H3YTP2R4ZFANCNFSM4HXOKHIQ .

jeff-zucker commented 5 years ago

rewrite the FolderGenerator I've added to work with all those testing environments.

Probably a good idea eventually. I need to finish tidying up solid-rest and add more tests for SolidFileClient unless you can think of other things I should prioritize, in which case, feel free to suggest. I think making sure that solid rest is dependable is a necessary pre-requisite to everything else. I think we can switch the tests to app: space now. But then we can test agains the others as needed. No reason to do all three every time.

jeff-zucker commented 5 years ago

This is unrelated to testing or to solid-file-client but I need to get it taken care of before I can continue. I'm confused about npm versioning. I had move to a 1.0.0 numbering scheme and rdflib which has ^0.1.12 in the dependency list does NOT pick it up (i.e. a fresh npm install gets 0.1.14, not anything in th 0.2 series I had or the 1.0 series. Do I need to stop using the 1.0 series and stick with 0.1?

And a second thing - I have the file and ls support in solid-rest/src/file.js and solid-rest/src/localStorage.js, not in their own repos. So in scripts I call them with require('solid-rest/file.js") which seems to work fine. This hugely simplifies things for me. Otherwise I have four packages to update if it's something impacting solid-auth-cli or five for solid-file-client, or six levels of libraries if its solid-ide. Will this bite me later?

Otto-AA commented 5 years ago

I need to finish tidying up solid-rest and add more tests for SolidFileClient unless you can think of other things I should prioritize, in which case, feel free to suggest.

I think the main priority (for solid-file-client) now is to finish the testing environment. This includes a test-fetch method and a folder structure generator which both have a global setting for switching between in-memory-tests, https, etc. I'd also create/move some utility methods for testing so they can be imported from all tests (e.g. resolvesWithStatus(promise, status), ...). I would finish this before adding more tests. Are you already working on these? If not, I could start tomorrow.

Otto-AA commented 5 years ago

I agree with you, that we don't need to always test in all environments. I'd stick with an in-memory solid-rest pod. Is this what you mean with "app: space"?

Otto-AA commented 5 years ago

Regarding the npm stuff, I've never published something there yet. I've seen the use of require('solid-rest/file.js') also in other projects, but I don't know what advantages/disadvantages it has. I also don't know much about versioning.

jeff-zucker commented 5 years ago

Yes, use app://ls as the address space for in-memory processing.

On Sat, Jun 15, 2019 at 10:27 AM A_A notifications@github.com wrote:

I agree with you, that we don't need to always test in all environments. I'd stick with an in-memory solid-rest pod. Is this what you mean with "app: space"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jeff-zucker/solid-file-client/issues/52?email_source=notifications&email_token=AKVJCJDBDVCLKXVZ6HO3WELP2UQ7XA5CNFSM4HXOKHI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY4QXY#issuecomment-502384735, or mute the thread https://github.com/notifications/unsubscribe-auth/AKVJCJF4YYBAPDQHOHRM3HDP2UQ7XANCNFSM4HXOKHIQ .

Otto-AA commented 5 years ago

Afaik localStorage isn't available in jest tests. I think we would need app://mem for that (in memory storage, just an object). Or mock localStorage, but that doesn't seems even more work.

jeff-zucker commented 5 years ago

Look at all.js and you'll see It uses app://ls/* i.e. localStorage which like file:// is now loaded by default with solid-rest [correction, it is loaded by default with solid-auth-cli, NOT with solid-rest]. I made an in-memory equivalent for non-browser use so it mocks setItem etc.

So localStorage is fully functional, has been tested to exactly correspond to the file: and https: results for basic REST.

jeff-zucker commented 5 years ago

I think the main priority (for solid-file-client) now is to finish the testing environment. This includes a test-fetch method and a folder structure generator which both have a global setting for switching between in-memory-tests, https, etc. I'd also create/move some utility methods for testing so they can be imported from all tests (e.g. resolvesWithStatus(promise, status), ...). I would finish this before adding more tests. Are you already working on these? If not, I could start tomorrow.

I would appreciate if you could do that part. Thanks so much!

bourgeoa commented 5 years ago

I think this is where you find the explanation why rdflib accept only minor changes in npm version https://docs.npmjs.com/about-semantic-versioning

jeff-zucker commented 5 years ago

Thanks, Alain, I'd read that. I also read but can't find right now an article about how the major release 0 cna mess things up so that ^0.1.12 can be problematic. So I am wondering if there is any way to get ^0.1.12 to pick up 1.0.6 or do I need to get rdflib to change to ^1.0.0 or *?

bourgeoa commented 5 years ago

I have been through the tests and the test API with POST for create folder do not work with filename cannot end with '/' I went to look at differences between POST and PUT. May be I misunderstand but it seems that POST should not be used to create a known URI like container but ressource and location contains the name of the created ressource

If my understanding is correct then the test should not succeed.

By the way @otto-AA many thanks for your jest explanations.

jeff-zucker commented 5 years ago

I'm sorry, can you explain what you mean about the test. It does not give the warning if the slug has a trailing slash or it does give the warning? A slash belongs in a pathname which is the first parameter to POST, but it does not belong in a filename which is the slug setting of POST. So the server should return 400 error if the user sends a slash in the slug.

The spec says that using PUT on a container is not valid so POST is all that we can use. NSS returns location for POST but does not contain it for PUT.

What action are you suggesting?

bourgeoa commented 5 years ago

I suppose I misinterpret something. Here is the test result : 

 ● core methods › creators › post resolves with 201 creating a new folder

    expect(received).resolves.toHaveProperty()

    Received promise rejected instead of resolved
    Rejected to value: {"body": undefined, "headers": {Symbol(map): {"allow": ["OPTIONS, HEAD, GET, PATCH, POST, PUT, DELETE"], "content-type": ["text/turtle"], "date": ["2019-06-15T15:36:39.215Z"], "link": ["<.meta>; rel=\"describedBy\", <.acl>; rel=\"acl\",<http://www.w3.org/ns/ldp#Container>; rel=\"type\",<http://www.w3.org/ns/ldp#BasicContainer>; rel=\"type\""], "location": ["file:///volume1/homes/admin/github/jeff-zucker/solid-file-client/test-folder/SolidApi/core/post/"], "x-powered-by": ["solid-rest-file-storage-1.0.0"]}}, "json": [Function bound _json], "ok": false, "status": 400, "statusText": "slashes not allowed in filenames", "text": [Function bound _text]}
jeff-zucker commented 5 years ago

Ah, no you didn't misinterpret. That test needs to be changed to expect

  1. It gets the correct response, it just doesnt expect it.

On Sat, Jun 15, 2019 at 1:43 PM Alain Bourgeois notifications@github.com wrote:

I suppose I misinterpret something. Here is the test result :

● core methods › creators › post resolves with 201 creating a new folder

expect(received).resolves.toHaveProperty()

Received promise rejected instead of resolved

Rejected to value: {"body": undefined, "headers": {Symbol(map): {"allow": ["OPTIONS, HEAD, GET, PATCH, POST, PUT, DELETE"], "content-type": ["text/turtle"], "date": ["2019-06-15T15:36:39.215Z"], "link": ["<.meta>; rel=\"describedBy\", <.acl>; rel=\"acl\",<http://www.w3.org/ns/ldp#Container>; rel=\"type\",<http://www.w3.org/ns/ldp#BasicContainer>; rel=\"type\""], "location": ["file:///volume1/homes/admin/github/jeff-zucker/solid-file-client/test-folder/SolidApi/core/post/"], "x-powered-by": ["solid-rest-file-storage-1.0.0"]}}, "json": [Function bound _json], "ok": false, "status": 400, "statusText": "slashes not allowed in filenames", "text": [Function bound _text]}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jeff-zucker/solid-file-client/issues/52?email_source=notifications&email_token=AKVJCJDKSDTNTJBDHR6PGV3P2VH67A5CNFSM4HXOKHI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXY7Y7A#issuecomment-502398076, or mute the thread https://github.com/notifications/unsubscribe-auth/AKVJCJG2WDGG7XO57432LITP2VH67ANCNFSM4HXOKHIQ .

bourgeoa commented 5 years ago

How do you create a folder then. I am missing something if POST return 400 and PUT is not allowed.

bourgeoa commented 5 years ago

As for npm versioning, I found https://semver.npmjs.com/

jeff-zucker commented 5 years ago

This is how a POST is made:

 return await fetch( pathToContainingFolder,{
    method:"POST",
    headers:{slug:nameOfItemCreated,link:link},
    body:content
  }

The pathToContainingFolder is a path, therefore it may contain slashes. The nameOfItemCreated is a file name (whether of a container or a resource) and may not contain slashes. So post( /foo/bar/, baz) creates /foo/bar/baz with baz's being a container or a resource depending on the link. BUT post(/foo/bar/,baz/) - note the trailing slash on baz/ throws a 400 error. If I'm still not clear, feel free to ask again.

jeff-zucker commented 5 years ago

I did not know any of this about the 400 error until I tried to run our tests against NSS rather than with file:// URLs.

jeff-zucker commented 5 years ago

As for npm versioning, I found https://semver.npmjs.com/

YES! Thank you, that validates my sanity. If you put ^0.1.12 in, there are no matches. How very odd. So I will need to ask for rdflib to put in ^1.0.0 or similar. You are really very good at finding out these obscure details. Thank you very much.

jeff-zucker commented 5 years ago

Afaik localStorage isn't available in jest tests.

Yes, it works fine. I've changed both of my crud tests to now work in memory using app://ls. The only real change was a getBase method which you may want to use for your test framework. In terms of setup, I really don't need anything but the existence of an empty test-folder ... which I take care of in a brief BeforeAll so not much needs to be done in advance for my tests.

jeff-zucker commented 5 years ago

I've verified that my crud tests in jest all pass in memory, on disk, and remotely.

bourgeoa commented 5 years ago

@Otto-AA I made some tests (tests succeeded) https://github.com/bourgeoa/solid-file-client/commit/5ed2b57f7dd796e0675ce34f2b7126b1ed2414f7 If you agree I can push them to jeff-zucker otto-changes.

There is a TODO test for SolidApi can you detail a bit the list so I can work on

jeff-zucker commented 5 years ago

@Otto-AA wrote (in chat)

I'm currently working on the test environment which uses app://ls/ and I'm running into issues with creating folders.

Have you installed the latest solid-rest? I made it so the / folder always exists which solves some of those issues. Other than that, there should be no difference between app://ls and any other URL. If you find one, let me know and I'll fix it in solid-rest

As I've already mentioned some time ago, I think a POST request to a non-existing folder-url shouldn't return 404.

But that is a very specific part of the spec and is how NSS behaves. I do not think we should behave differently with a basic method like post. If we want different behavior, it should be in the createFolder method, not in post.

Because of this the createFolder method fails

You can't trap the error? One thing you may want to do, a bit of a kludge but effective, transparent and it works: if you need to create folders recursively, instead of doing multiple posts on the intermediary folders, do PUT of a blank .meta file on the deepest folder. So PUT(/foo/bar/baz/.meta) will create the folder tree.

That's the code I'm referring to: https://github.com/jeff-zucker/solid-rest/blob/master/src/rest.js#L82

Well, yes I made solid-rest behave according to the spec, what is the problem? Do you seriously want our package to behave differently from the spec at this basic level? Why?

And a little suggestion for solid-rest: ... the options object has been modified.

Yes, I use it to pass along information picked up along the way (such as the original requested schem, whether we're looking at a container or not, whether the item pre-exists, etc. which can be use down the line. Perhaps that's not the best way to do it. I'm unclear on the specifics of what you're suggesting.

jeff-zucker commented 5 years ago

Just to make clear, POST does not return 404 if the folder being posted doesn't exist - it returns it if the parent the folder is being posted to doesn't exist. The 404 in solid-rest's POST is for the parent folder, not the folder being created.

Otto-AA commented 5 years ago

Just to make clear, POST does not return 404 if the folder being posted doesn't exist - it returns it if the parent the folder is being posted to doesn't exist. The 404 in solid-rest's POST is for the parent folder, not the folder being created.

That's exactly the behavior I expected but isn't what I perceived. I didn't want to tell you to diverge from the spec, but rather to follow it (because I perceived it didn't). I assumed that app://ls/ is already existing and a post to it with the slug "test-folder" should succeed. But I got a 404, hence I tracked it a bit down and found out that objectExists was equal to false. That's why I assumed this would be the error.

I will take a look at it again later today/tomorrow. If I still encounter this issue with the newest versions (which I think I already used but I'll double check it), I'll give you clear steps to reproduce the issue. If not I'll tell you that it was something different.

jeff-zucker commented 5 years ago

Ah ,OK, there could be a bug in object-exists, I'll look for that.

jeff-zucker commented 5 years ago

BTW, haven't committed yet, but I am adding a getStorage and setStorage method to solid-auth-cli. This will let users replace the default storages if wanted and also to directly use methods the storage handler provides. So for example I made a dump() method in localStorage which shows all current files and folders in storage. So this works:

const auth = require("solid-auth-cli")
const storage = auth.getStorage("ls")
storage.dump()