search

yawik / Solr

This YAWIK module enables solr search for job openings.
http://jobs.yawik.org
MIT License
1 stars 2 forks source link

Fetching HTML source if no description is given #12

Closed cbleek closed 6 years ago

cbleek commented 6 years ago

@fedys @TiSiE

adding a jobosting without a fulltext to solr makes no sense. So what can we do, if a job is activated and no html is given.

My idea is to simply fetch the html. If fetching fails, the job is not inserted.

Can we/should we do this in the solr module?

Keep in mind that all the time consuming tasks like fetching pages, inserting into solr, sending mails... should be moved into some queing system

fedys commented 6 years ago

I am not sure if the Solr module should be responsible for fetching an html. In my opinion the code which inserts an active Job should be responsible for a data completeness. If there is no description field in the import specification I would fetch an html in the SimpleImport module.

cbleek commented 6 years ago

@TiSiE do you agree?

TiSiE commented 6 years ago

Yes, I do agree.

cbleek commented 6 years ago

this will be done in yawik/SimpleImport#4