search

ooyala / spark-jobserver

REST job server for Spark. Note that this is *not* the mainline open source version. For that, go to https://github.com/spark-jobserver/spark-jobserver. This fork now serves as a semi-private repo for Ooyala.
Other
344 stars 135 forks source link

New Jar Files With Same App Name Not Used #91

Open paulmagid opened 9 years ago

paulmagid commented 9 years ago

If you submit a new jar file to a named context only the first Jar file is executed. Subsequent uploads are ignored until a spark-jobserver restart. If you submit a jar to the default context this issue does not occur.

velvia commented 9 years ago

This is expected. Right now due to the way classloaders work, you cannot update the same jar in the same context. However, you don't have to restart the jobserver. You can simply -X DELETE the context and when you create a new one the jar will reload.

Enabling dynamic reload of the same jar in the same context is a pretty difficult problem and would require more changes in Spark itself.

On Mon, Nov 24, 2014 at 10:18 AM, paulmagid notifications@github.com wrote:

If you submit a new jar file to a named context only the first Jar file is executed. Subsequent uploads are ignored until a spark-jobserver restart. If you submit a jar to the default context this issue does not occur.

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/91.

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

paulmagid commented 9 years ago

From looking at the code this is what we expected might be the case: that jobserver behaves this way by design and the issue cannot be easily fixed due to other upstream problems.

Would it be possible to treat this as a documentation bug because this cost us 5 days to track down what was causing the behavior then narrow in and read the code and really understand what is going on? We could not find documentation or reports of this issue/behavior anywhere. Maybe we were not looking in the right spots but probably the behavior and the work-around should be clearly documented somewhere in the official documentation.

velvia commented 9 years ago

@paulmagid, sorry about the wasted time. Yes, let's definitely make it a doc bug.

On Tue, Nov 25, 2014 at 7:46 AM, paulmagid notifications@github.com wrote:

From looking at the code this is what we expected might be the case: that jobserver behaves this way by design and the issue cannot be easily fixed due to other upstream problems.

Would it be possible to treat this as a documentation bug because this cost us 5 days to track down what was causing the behavior then narrow in and read the code and really understand what is going on? We could not find documentation or reports of this issue/behavior anywhere. Maybe we were not looking in the right spots but probably the behavior and the work-around should be clearly documented somewhere in the official documentation.

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/91#issuecomment-64419703 .

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

velvia commented 9 years ago

Speaking of which, would it be helpful to create a wiki instead of a giant README?

On Tue, Nov 25, 2014 at 8:34 AM, Evan Chan velvia@gmail.com wrote:

@paulmagid, sorry about the wasted time. Yes, let's definitely make it a doc bug.

On Tue, Nov 25, 2014 at 7:46 AM, paulmagid notifications@github.com wrote:

From looking at the code this is what we expected might be the case: that jobserver behaves this way by design and the issue cannot be easily fixed due to other upstream problems.

Would it be possible to treat this as a documentation bug because this cost us 5 days to track down what was causing the behavior then narrow in and read the code and really understand what is going on? We could not find documentation or reports of this issue/behavior anywhere. Maybe we were not looking in the right spots but probably the behavior and the work-around should be clearly documented somewhere in the official documentation.

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/91#issuecomment-64419703 .

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

paulmagid commented 9 years ago

I think so… At some point the amount of info gets too unwieldy for a README.

Thanks,

Paul Magid Toyota Motor Sales IS Enterprise Architecture (EA) Architect I R&D Ph: 310-468-9091 (X69091) PCN 1C2970, Mail Drop PN12

From: Evan Chan [mailto:notifications@github.com] Sent: Tuesday, November 25, 2014 8:35 AM To: ooyala/spark-jobserver Cc: Paul Magid Subject: Re: [spark-jobserver] New Jar Files With Same App Name Not Used (#91)

Speaking of which, would it be helpful to create a wiki instead of a giant README?

On Tue, Nov 25, 2014 at 8:34 AM, Evan Chan velvia@gmail.com<mailto:velvia@gmail.com> wrote:

@paulmagid, sorry about the wasted time. Yes, let's definitely make it a doc bug.

On Tue, Nov 25, 2014 at 7:46 AM, paulmagid notifications@github.com<mailto:notifications@github.com> wrote:

From looking at the code this is what we expected might be the case: that jobserver behaves this way by design and the issue cannot be easily fixed due to other upstream problems.

Would it be possible to treat this as a documentation bug because this cost us 5 days to track down what was causing the behavior then narrow in and read the code and really understand what is going on? We could not find documentation or reports of this issue/behavior anywhere. Maybe we were not looking in the right spots but probably the behavior and the work-around should be clearly documented somewhere in the official documentation.

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/91#issuecomment-64419703 .

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

— Reply to this email directly or view it on GitHubhttps://github.com/ooyala/spark-jobserver/issues/91#issuecomment-64428677.