search

101companies / 101worker

4 stars 11 forks source link

All of 101repo must be processed by 101worker again #195

Closed rlaemmel closed 9 years ago

rlaemmel commented 9 years ago

cc @kevin-klein @martinleinberger

This is urgent.

It seems 101worker, these days, only processes "contributions". However, there is important source code also in the "technologies", "languages", and "concepts" folders. In fact, 101wiki fails to properly work on related pages without those folders being processed (e.g., in terms of fact extraction).

This relates to this 101wiki issue, for example:

https://github.com/101companies/101rails/issues/65

The relevant folders are all known due to the 101wiki page-to-repo links. Please take them into account.

@hartenfels , if you have any questions, please get in touch with @kevin-klein or @martinleinberger on the usual channels.

hartenfels commented 9 years ago

101worker is processing those folders fine. See for example this concept.

The issue is not that the folders aren't being processed, it's that whatever module is supposed to process them is not working. It probably didn't work for a long time and nobody noticed because the files remained until worker was reset.

rlaemmel commented 9 years ago

This was working with the old worker (pull101repo and various other modules). The issue lies in how the 101wiki-pages-to-101repo-links need to completely covered by the talks of the modules.

hartenfels commented 9 years ago

The bug isn't that 101worker is not processing all folders though, because it clearly is. The issue is somewhere else.

rlaemmel commented 9 years ago

Good point, Carsten. For instance, I just checked that modules were applied to Hello World: http://data.101companies.org/resources/concepts/Hello_world_program/ Thus, the issue may be in the services of 101explorer. Re-assigning to @martinleinberger in the hope he can take over from here.

rlaemmel commented 9 years ago

Also, @kevin-klein, if you can add any insight on the nature of https://github.com/101companies/101rails/issues/65 please do so there or here.

kevin-klein commented 9 years ago

Considering these files the required files seem to exist: http://worker.101companies.org/data/resources/concepts/Local_scope/

But i think members is and was the issue much more which did not work on other files than contributions and now not at all. I will have a look.

kevin-klein commented 9 years ago

The issue was a pessimistic clean up of languageExtractor module.