Register project: Documentation Readers

davidediruscio commented 4 years ago

Hello,

I have finished the last elements for the documentation readers and they can be now added to the platform UI.

More specifically there are two (three depending on the capacities of the UI) types of documentation:

Git-Based (org.eclipse.scava.repository.model.documentation.gitbased). The requirements are the same as a Git project (ony URL), as it is based on it.
Systematic (org.eclipse.scava.repository.model.documentation.systematic). There are two types of documentation systematic:
- No login necessary. In this aspect there are only two mandatory fields: URL and EXECUTIONFREQUENCY
- Login necessary. For a documentation behind a login, we need more fields that are mandatory: URL, EXECUTIONFREQUENCY, LOGINURL, USERNAME, USERNAMEFIELDNAME, PASSWORD and PASSWORDFIELDNAME.

As you can see for the documentation systematic, if no login is necessary only two fields must be filled, but if a login is necessary, then all the fields are mandatory. So, I don't know if it will be easier to create two types of Systematic one with login and one without login. That's up to you and the capacities of the UI.

Apart from that there is one more limitation, the gitbased documentation cannot be used along with a git project, because internally they use the same model and the documentation extends the reader of the project.

davidediruscio commented 4 years ago

@creat89, the commit 2cbbe67 should add a support to the documentation readers.

davidediruscio commented 4 years ago

Hello @ambpro, for Documentation Git-Based I get the following issue when I create a project:

oss-app_1         | Adding new project
oss-app_1         | Jun 18, 2019 1:46:44 PM org.restlet.engine.log.LogFilter afterHandle
oss-app_1         | INFO: 2019-06-18    13:46:44        172.28.0.10     -       172.28.0.8      8182    POST    /projects/create        -       201     1146    248     64      http://oss-app:8182     Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0     http://192.168.42.216:5601/
oss-app_1         | Error: null
oss-app_1         | null
oss-app_1         | Jun 18, 2019 1:46:44 PM org.restlet.resource.ServerResource doCatch
oss-app_1         | WARNING: Exception or error caught in server resource
oss-app_1         | Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:540)
oss-app_1         |     at org.restlet.resource.ServerResource.get(ServerResource.java:743)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:618)
oss-app_1         |     at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:679)
oss-app_1         |     at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:357)
oss-app_1         |     at org.restlet.resource.ServerResource.handle(ServerResource.java:1044)
oss-app_1         |     at org.restlet.resource.Finder.handle(Finder.java:236)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:75)
oss-app_1         |     at org.restlet.Application.handle(Application.java:385)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.Component.handle(Component.java:408)
oss-app_1         |     at org.restlet.Server.handle(Server.java:507)
oss-app_1         |     at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
oss-app_1         |     at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
oss-app_1         |     at org.restlet.ext.jetty.JettyServerHelper$WrappedServer.handle(JettyServerHelper.java:273)
oss-app_1         |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
oss-app_1         |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
oss-app_1         |     at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
oss-app_1         |     at java.lang.Thread.run(Thread.java:748)
oss-app_1         | Caused by: java.lang.NullPointerException
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.escape(Util.java:52)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessage(Util.java:45)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessageRepresentation(Util.java:33)
oss-app_1         |     at org.eclipse.scava.platform.client.api.ProjectListResource.doRepresent(ProjectListResource.java:92)
oss-app_1         |     at org.eclipse.scava.platform.client.api.AbstractApiResource.represent(AbstractApiResource.java:55)
oss-app_1         |     at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
oss-app_1         |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
oss-app_1         |     at java.lang.reflect.Method.invoke(Method.java:498)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:524)
oss-app_1         |     ... 47 more

After, if I want to add or see the projects page I get the following issue:

oss-app_1         | Error: null
oss-app_1         | null
oss-app_1         | Jun 18, 2019 1:49:20 PM org.restlet.resource.ServerResource doCatch
oss-app_1         | WARNING: Exception or error caught in server resource
oss-app_1         | Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:540)
oss-app_1         |     at org.restlet.resource.ServerResource.get(ServerResource.java:743)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:618)
oss-app_1         |     at org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:679)
oss-app_1         |     at org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:357)
oss-app_1         |     at org.restlet.resource.ServerResource.handle(ServerResource.java:1044)
oss-app_1         |     at org.restlet.resource.Finder.handle(Finder.java:236)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.engine.application.ApplicationHelper.handle(ApplicationHelper.java:75)
oss-app_1         |     at org.restlet.Application.handle(Application.java:385)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Router.doHandle(Router.java:422)
oss-app_1         |     at org.restlet.routing.Router.handle(Router.java:639)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.engine.application.StatusFilter.doHandle(StatusFilter.java:140)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.routing.Filter.doHandle(Filter.java:150)
oss-app_1         |     at org.restlet.routing.Filter.handle(Filter.java:197)
oss-app_1         |     at org.restlet.engine.CompositeHelper.handle(CompositeHelper.java:202)
oss-app_1         |     at org.restlet.Component.handle(Component.java:408)
oss-app_1         |     at org.restlet.Server.handle(Server.java:507)
oss-app_1         |     at org.restlet.engine.connector.ServerHelper.handle(ServerHelper.java:63)
oss-app_1         |     at org.restlet.engine.adapter.HttpServerHelper.handle(HttpServerHelper.java:143)
oss-app_1         |     at org.restlet.ext.jetty.JettyServerHelper$WrappedServer.handle(JettyServerHelper.java:273)
oss-app_1         |     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
oss-app_1         |     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
oss-app_1         |     at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
oss-app_1         |     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
oss-app_1         |     at java.lang.Thread.run(Thread.java:748)
oss-app_1         | Caused by: java.lang.NullPointerException
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.escape(Util.java:52)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessage(Util.java:45)
oss-app_1         |     at org.eclipse.scava.platform.client.api.Util.generateErrorMessageRepresentation(Util.java:33)
oss-app_1         |     at org.eclipse.scava.platform.client.api.ProjectListResource.doRepresent(ProjectListResource.java:92)
oss-app_1         |     at org.eclipse.scava.platform.client.api.AbstractApiResource.represent(AbstractApiResource.java:55)
oss-app_1         |     at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
oss-app_1         |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
oss-app_1         |     at java.lang.reflect.Method.invoke(Method.java:498)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:524)
oss-app_1         |     ... 47 more
oss-app_1         |

EDIT: Interesting the project was added correctly to MongoDB

davidediruscio commented 4 years ago

By looking on the stack trace, I guess that the above issue is already fixed by the commit https://github.com/crossminer/scava/commit/7e232944198e100368ab0de9037337fcd757f8db. let me know if the current update of dev branch works fine for you?

davidediruscio commented 4 years ago

Ok, I'll build again the platform once the build will be finished

davidediruscio commented 4 years ago

Hello,

Checking the UI I have seen these issues for the documentation systematic with login:

where Password Field Name is considered as a password rather than text.

And, Martin and I, we're seeing that the data is not being passed to Mongo DB, either with or without login.

davidediruscio commented 4 years ago

@creat89 I pushed a fix ( commit 7d7fa48) to handle the above issue. Could you check it again in your side?

davidediruscio commented 4 years ago

Hello @ambpro,

When I create a Documentation Systematic I get the following error:

oss-app_1         | Internal Server Error (500) - The server encountered an unexpected condition which prevented it from fulfilling the request
...
oss-app_1         | Caused by: java.lang.NullPointerException
oss-app_1         |     at org.eclipse.scava.platform.client.api.ProjectCreationResource.createProject(ProjectCreationResource.java:195)
oss-app_1         |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
oss-app_1         |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
oss-app_1         |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
oss-app_1         |     at java.lang.reflect.Method.invoke(Method.java:498)
oss-app_1         |     at org.restlet.resource.ServerResource.doHandle(ServerResource.java:521)
oss-app_1         |     ... 45 more

Thus, nothing is stored in Mongo.

And still, I get the password dots in the password field, despite you have changed the file (I have checked and built the platform some minutes ago and used the incognito mode). So, I don't know what's going on.

davidediruscio commented 4 years ago

@creat89

When I create a Documentation Systematic I get the following error: Thus, nothing is stored in Mongo.

The commit (7d28be0) should fix that. Now, I am able to register a project with a Documentation Systematic connector.

And still, I get the password dots in the password field, despite you have changed the file (I have checked and built the platform some minutes ago and used the incognito mode).

I am pretty sure that it's a browser cache problem! can you try Ctrl + F5 on your browser and let me know if it works?

davidediruscio commented 4 years ago

Hello @ambpro,

The connector works well, but the data is missing for the Systematic Documentation without login. I'm getting this: Instead of something like this:

And the Password field is still showing like password despite using Ctrl+F5 and incognito mode.

davidediruscio commented 4 years ago

Hello @creat89

The connector works well, but the data is missing for the Systematic Documentation without login. I'm getting this

It's a bit strange because with the current metric-platform build, I can add a new documentation systematic through the Register project . with no-login measure-project-wiki

with login

davidediruscio commented 4 years ago

@ambpro with the last build:

No login: If execution frequency field is filled, the data is stored in Mongo. However, we can still click on save without indicating the frequency. If this is the case the entry is not stored.
Login: If execution frequency field is filled, the data is stored in Mongo. We can still click on save without indicating the frequency. However, once we click on save, the button becomes gray and it is impossible to make it active until we delete the project. Filling the execution frequency field does not change anything. The password field name is still showing dots.

davidediruscio commented 4 years ago

@creat89 , does the previous commit (c57a4a0) solve the problem ?

davidediruscio commented 4 years ago

Hello @ambpro,

In some cases yes, in some cases no. These are my observations: No login:

Currently, the project is never saved correctly, whether we fill the frequency field or not. In fact it is storing only the data shown in https://github.com/crossminer/scava/issues/252#issuecomment-537427108.

Login:

The project is saved correctly in MongoDB even if the frequency field is empty.
The password field name is still showing the dots.
The project can be saved without the password and username fields-related completed.

davidediruscio commented 4 years ago

@ambpro, I don't know if this information will make easier to fix the issue, but if the username and password fields are empty, the reader consider that automatically as without login. This could make easier to fix the issue, as currently the creation of the project with login stores the data correctly even if these fields are empty.

davidediruscio commented 4 years ago

@creat89

Systematic (org.eclipse.scava.repository.model.documentation.systematic). There are two types of documentation systematic:

No login necessary. In this aspect there are only two mandatory fields: URL and EXECUTIONFREQUENCY

Login necessary. For a documentation behind a login, we need more fields that are mandatory: URL, EXECUTIONFREQUENCY, LOGINURL, USERNAME, USERNAMEFIELDNAME, PASSWORD and PASSWORDFIELDNAME.

No login: If execution frequency field is filled, the data is stored in Mongo. However, we can still click on save without indicating the frequency. If this is the case the entry is not stored.

Login: If execution frequency field is filled, the data is stored in Mongo. We can still click on save without indicating the frequency.

I don't know if this information will make easier to fix the issue, but if the username and password fields are empty, the reader consider that automatically as without login.

AFAIU and based on the documentation reader spec, all the fields are mandatory depend on the reader type option (no login- with login). Is it still the case? Otherwise, could you clarify this point with more details?

davidediruscio commented 4 years ago

It's correct what you're saying, if you set a username and password you need to set the login URL and the username field and the password field. Otherwise the platform will throw an error indicating that you're missing some fields. However, what I'm trying to say is that you can create an object with the username and password fields empty and that would be equivalent to a no-login. Is that clear?

davidediruscio commented 4 years ago

However, what I'm trying to say is that you can create an object with the username and password fields empty and that would be equivalent to a no-login. Is that clear?

Ok. I'll adjust the reader's saving process and I will let you know asap!

davidediruscio commented 4 years ago

@creat89 I pushed some fixes related to what we discussed earlier. Can you re-test it again?

davidediruscio commented 4 years ago

Hello @ambpro,

No login: No matter whether the execution frequency is set or not, the saved object in MongoDB is still wrong. Like in https://github.com/crossminer/scava/issues/252#issuecomment-537427108
Login: Everything is stored correctly whether the user specifies an execution frequency or not. So this is ok, except for the password field name, which it is still showing as password.

I don't know if I'm seeing truly the last version. I'm using the latest version of the 11th Nov 2019 and using Firefox on a private windows to prevent the cache. Here is a screen:

davidediruscio commented 4 years ago

Hello @creat89 Related to cache issue, there you go a screenshot of the documentation-systematic template form from a fresh docker instance deployed this morning!

I see that the screenshot you have attached is not 100% up-to-date as the mine. Do you build the docker images using: docker-compose build --no-cache as mentionned in here ?

davidediruscio commented 4 years ago

I'll deploy again the newest version, but my question would be, how can we ensure that the cache is removed? I'm using a private window, which shouldn't store the cache. I have just tried Internet Explorer and it is the same. I guess it is something related on the docker side...

davidediruscio commented 4 years ago

I'll deploy again the newest version, but my question would be, how can we ensure that the cache is removed? I'm using a private window, which shouldn't store the cache. I have just tried Internet Explorer and it is the same. I guess it is something related on the docker side...

Yes! It seems that it's related to a docker images caching and it's not related to the browser you 're using!

The command docker-compose build --no-cache should deal with that (I hope that :) )

davidediruscio commented 4 years ago

I'm sure you'll not believe but still nothing :/

The weird thing is that I always use docker-compose build --no-cache and before that I have even done a system prune.

davidediruscio commented 4 years ago

@aabherve could you have a check on your side about this issue ? Thanks

davidediruscio commented 4 years ago

Hi @creat89 , you are using which branch of scava-deployment? If you are using the dev brach you should view the download and compilation of the web-admin code when you do the build.

Try running docker-compose build --no-cache admin-webapp. Do you see the building of node.js code?

davidediruscio commented 4 years ago

I don't think. Because it seems that ther code for the web-admin is the old one. I'll update it and see what happens again. In all the cases, I only see this:

davidediruscio commented 4 years ago

Yes, everything seems to work fine and as expected. The missing lines in the docker were the problem. I'll close the issue and thanks @ambpro and @MarcioMateus

davidediruscio commented 4 years ago

Hello,

I'm having this issue and I don't know what's the problem or what field is wrong:

davidediruscio commented 4 years ago

So, sometimes when you click wrongly the login necessary, some of the fields are expected to be filled, but those are hidden.

ambpro commented 4 years ago

The last commit (2eb03eb) provides some UI enhancements and form validation fixes to address the issue at https://github.com/eclipse-researchlabs/scava/issues/28#issuecomment-571205901. Please, test it and provide me feedback!

doc-sys

eclipse-researchlabs / scava

Register project: Documentation Readers #28