IOS lockup, interaction of sound with pointer tracking.

[ddyer0]

I can produce this with some regularity, but not instantly. The symptom is that the application completely locks, which makes it rather difficult. Running under the simulator in xcode, I find the following:

thread 1 (responding to pointer dragged) waiting for "display lock" held by thread 3 thread 2 (gc) waiting for thread 9 thread 3 (main edt thread) blocked by gc while acquiring "display lock" thread 9 (playing a sound clip) blocked by gc, stuck in native code

so nothing will happen until thread 9 proceeds, which doesn't seem to be happening. I'm guessing the problem is related to the state of thread 1. In the screen show I've opened the stack trace for those threads.

[ddyer0]

Supporting evidence for my interpretation - if I turn off the sound (the app no longer plays sounds) I haven't been able to produce the problem. Also, I don't think this is a new problem, but it's become much worse recently because I am working the garbage collector a lot harder.

[shannah]

Can you post the stack trace for thread 9 from your previous post?

On Tue, Jul 2, 2019 at 1:15 PM ddyer0 notifications@github.com wrote:

Supporting evidence for my interpretation - if I turn off the sound (the app no longer plays sounds) I haven't been able to produce the problem. Also, I don't think this is a new problem, but it's become much worse recently because I am working the garbage collector a lot harder.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/codenameone/CodenameOne/issues/2860?email_source=notifications&email_token=AAUNWOX7QMYWD2VH5RECLHDP5OZPZA5CNFSM4H45QBG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZCNQSQ#issuecomment-507828298, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUNWOXADS63ADDPTORJQO3P5OZPZANCNFSM4H45QBGQ .

-- Steve Hannah Web Lite Solutions Corp.

[ddyer0]

Codenameone thread 9 is the one labeled thread 27 in the left hand pane. You can see details in the right hand pane, inluding the threadID 9.

I have additional information - I'm able to produce the problem without pointer tracking or network, so there are just display, garbage creation, and sound clips going on. Pointer activity probably was involved only because it induced the other activity.

New theory (no supporting evidence yet) one of the sound clips is missing or malformed, causing unusual state in the sound playing thread.

[ddyer0]

I found at least part of the problem - I was calling EncodedImage.createFromImage(im.scaled(w,h),false) outside of the EDT. It's strange that I can't do image scaling as a background process, and there aren't any warnings in the documentation.

[ddyer0]

I found another instance of this behavior, this time involving a mode in which my app plays a lot of short sound clips. I found that changing clip.play to a Display.startInEdt seems to fix the problem.

[ddyer0]

Based on the new insight, this new test crashes after about 15 minutes on my Ipad, there's no reason why it should't run forever. To test this, I recommend plugged in with settings to never sleep.

package com.boardspace.dtest;

import java.io.IOException;
import java.io.InputStream;
import java.util.Hashtable;

import com.codename1.media.Media;
import com.codename1.media.MediaManager;
//
// issue #2989, layout not occurring
//
import com.codename1.ui.Component;
import com.codename1.ui.Container;

import com.codename1.ui.Display;
import com.codename1.ui.Form;
import com.codename1.ui.Graphics;
import com.codename1.ui.Image;
import com.codename1.ui.geom.Dimension;
import com.codename1.ui.layouts.Layout;
import com.codename1.ui.plaf.UIManager;
import com.codename1.ui.util.Resources;

class AudioClip {
    String name;
    InputStream data;
    String format = "audio/wav"; //"audio/wave";
    Media lastMedia = null;
    public AudioClip(String n,InputStream d)
    {   name = n;
        data = d;
    }
    public void play() {
        if(data!=null)
        {
            try {
                data.reset();               
                Media media = MediaManager.createMedia(data, format , 
                        new Runnable() { public void run() { 
                            lastMedia = null;
                        } });
                lastMedia = media;
                if(media!=null) { media.play(); }                
                }
            catch (IOException ex) 
                { 
                System.out.println("Trouble playing "+name+" : "+ex);
                }
        }
    }

}
class SoundManager implements Runnable 
{   private static SoundManager theInstance = null;
    private static Thread RAThread=null;
    private  boolean exit = false;
    private  int readPtr = 0;
    private  int writePtr = 0;
    static final int QUEUELENGTH = 20;
    private  String[] sounds = new String[QUEUELENGTH];
    private  long[] delays = new long[QUEUELENGTH];
    private  long nextDelay=0;
    private  Hashtable<String,AudioClip> soundClips = new Hashtable<String,AudioClip>();

    private String extractAClip()
        {
            long now = System.currentTimeMillis();

            try
            {

                while ((readPtr == writePtr) || (now<delays[readPtr]))
                {   if(readPtr==writePtr) 
                        { 
                          synchronized(this) { wait(); }
                        }
                        else 
                        { synchronized (this) { wait(Math.max(1, delays[readPtr]-now)); }};
                    now = System.currentTimeMillis();
                }
            }
            catch (InterruptedException err)
            { // pro forma catch
            }
            now = System.currentTimeMillis();
            if ((readPtr != writePtr) && now>=delays[readPtr])
            {
                String clipName = sounds[readPtr];
                readPtr++;
                if (readPtr == QUEUELENGTH)
                {
                    readPtr = 0;
                }

                return (clipName);
            }

            return (null);
        }

        private AudioClip getAClip()
        {   String n = extractAClip();
            return(GetCachedClip(n));
        }

        private AudioClip GetCachedClip(String clipUrl)
        {   if (clipUrl != null)
            {   String name = clipUrl;
                AudioClip clip = soundClips.get(name);

                if (clip == null)
                {   
                    clip = LoadAClip(clipUrl);

                    if (clip != null)
                    {
                        soundClips.put(name, clip);
                    }
                }
                return (clip);
            }
            return (null);
        }

        private InputStream openStream(String name) throws IOException 
        {   
            return(Dtest.theme.getData(name));
        }

        private AudioClip getAudioClip(String url) {
            InputStream s;
            try {
                s = openStream(url);
                if(s!=null) { return(new AudioClip(url,s)); }
            } catch (IOException e) {
                Dtest.postError("Missing AudioClip "+url);
            }
            Dtest.postError("Missing AudioClip "+url);
            return(null);
        }
        private AudioClip LoadAClip(String name)
        {
            AudioClip v = null;

            try
            {   
                v = getAudioClip(name);
            }
            catch (NullPointerException ex)
            {
                 /* shouldn't happen */
            }
            catch (Throwable err)
            {
                Dtest.postError( "LoadAClip Failed for " + name + err);
            }

            return (v);
        }

        private synchronized static void makeNewInstance()
        {   if(theInstance==null)
            {
            theInstance = new SoundManager();
            RAThread = new Thread(theInstance,"RA thread");
            RAThread.start();
            }
        }
        private static SoundManager getInstance()
        {   if(theInstance==null)
                {
                makeNewInstance();
                }
            return(theInstance);
        }

        static void playASoundClip(String clip,int delay)
        {   
            getInstance().playAsoundClip(clip, delay);
        }

        private synchronized void playAsoundClip(String clip,int delay)
        {   long now = System.currentTimeMillis();
            sounds[writePtr] = clip;
            delays[writePtr] = nextDelay;
            nextDelay=Math.max(Math.min(now+5000,nextDelay),now)+delay;
            int nextwp = writePtr;
            nextwp++;
            if(nextwp==QUEUELENGTH) { nextwp = 0; }
            if(nextwp==readPtr) { System.out.println("sound queue full playing "+clip); }
            else { writePtr = nextwp; }

            notify(); 
        }

    public void run()
    {   
        try {
        //System.out.println("run");
        RAThread = Thread.currentThread();
        for (; !exit;)
        {   AudioClip clip = getAClip();;
            if (clip != null)
                {   clip.play();
                }
             };
        }
        catch (Throwable err)
        { Dtest.postError("in round manager run()"+err); 
        }
    }

    public static void stop()
    {   getInstance().stopNow();
    }
    private void stopNow()
    {   exit = true;
    }
}
interface NullLayoutProtocol {
    public void doNullLayout();
}

class NullLayout extends Layout 
{   
    NullLayoutProtocol expectedParent;
    public NullLayout(NullLayoutProtocol parent) { expectedParent = parent; }
    /* Required by LayoutManager. */
    public void addLayoutComponent(String name, Component comp)    {  }

    /* Required by LayoutManager. */
    public void removeLayoutComponent(Component comp)    {  }

    /* Required by LayoutManager. */
    public Dimension preferredLayoutSize(Container parent) {
        Dimension dim = new Dimension(parent.getWidth(), parent.getHeight());
        return dim;
    }

    /* Required by LayoutManager. */
    public Dimension minimumLayoutSize(Container parent) {
        Dimension dim = new Dimension(1, 1);
        return dim;
    }

    public void layoutContainer( Container parent) 
    {   System.out.println("layout "+this);
        if(expectedParent!=null) { expectedParent.doNullLayout(); }
    }
    public Dimension getPreferredSize(Container parent) { return(new Dimension(parent.getWidth(),parent.getHeight())); }

}

class ParentPanel extends Container implements NullLayoutProtocol
{
    public void doNullLayout() {
        setWidth(getParent().getWidth());
        setHeight(getParent().getHeight());
        for(int i=0;i<getComponentCount();i++)
        {
            Component c = getComponentAt(i);
            if(c instanceof NullLayoutProtocol)
            {
                ((NullLayoutProtocol)c).doNullLayout();
            }
        }}

}
class TestWindow extends Container implements NullLayoutProtocol
{
    public boolean laidout = false;
    public int paints = 0;
    public void doNullLayout()
        { 
          System.out.println("layout "+this);
          setWidth(getParent().getWidth());
          setHeight(getParent().getHeight());
          laidout = true; 
        }

    public void paint(Graphics g)
    {   int w = Math.max(500, getWidth());
        int h = Math.max(300, getHeight());
        Image offscreen = Image.createImage(w,h);
        Graphics offG = offscreen.getGraphics();

        offG.setColor(0x808080);
        offG.fillRect(0, 0, w, h);
        offG.setColor(0x0);
        paints++;
        if(laidout)
        {
            offG.drawString("Laid Out "+w+"x"+h+" :"+paints,0,h/2);
        }
        else
        {
            offG.drawString("NOT LAID OUT "+w+"x"+h+" :"+paints,0,h/2);
        }
        if(Dtest.error!=null)
        {
            offG.drawString(Dtest.error,0,h/4);
        }
        g.drawImage(offscreen,0,0);
    }
}
public class Dtest  {

    private Form current;
    @SuppressWarnings("unused")
    public static Resources theme;
    public void init(Object context) {
        theme = UIManager.initFirstTheme("/theme");
        // Pro only feature, uncomment if you have a pro subscription
        // Log.bindCrashProtection(true);
    }
    int loops = 0;
    TestWindow content = null;
    public void addContent(Container hi)
    {
        ParentPanel p = new ParentPanel();
        p.setLayout(new NullLayout(null)); 
        // set a default size so the display isn't blank.
        p.setWidth(500);
        p.setHeight(500);
        TestWindow w = content = new TestWindow();
        w.setLayout(new NullLayout(p));
        w.setVisible(true);
        // set a default size so the display isn't blank.
        w.setWidth(500);
        w.setHeight(500);
        hi.add(p);
        p.add(w);
        w.setVisible(true);

    }
    public static boolean isEdt()
    {
        Display dis = Display.getInstance();
        return(dis.isEdt());
    }
    public static void runInEdt(Runnable r)
    {   if(isEdt())
            {
            r.run();
            }
        else
            {
            Display.getInstance().callSeriallyAndWait(r); 
            }
    }
    static String error = null;
    static void postError(String msg)
    {
        error = msg;
    }
    @SuppressWarnings("deprecation")
    public void start() {
        new Thread(new SoundManager()).start();
        if(current != null){
                current.show();
                return;
            }

        Form hi = current = new Form("test form"); 
        // adding the content here fixes the problem
        // addContent(hi);
        //
        // curiously, this doesn't help
        //hi.setAllowEnableLayoutOnPaint(true);
        //
        //
        hi.show();
        addContent(hi);

        Runnable rr = new Runnable (){
            public void run() {

            System.out.println("running");
            while(true)
            {
            try {
            Thread.sleep(50);
            if(content==null)
                {
                runInEdt(new Runnable() { public void run() { addContent(hi); }});
                }
            SoundManager.playASoundClip("click.wav",50);
            content.repaint();

            }

              catch (InterruptedException e) {};
                }}};
            new Thread(rr).start();

    }

    public void stop() {
        current = (Form)Display.getInstance().getCurrent();
    }

    public void destroy() {
    }

}````
you'll need to include a small sound file in the theme.res file.  I've attached the theme.res I use
[theme.zip](https://github.com/codenameone/CodenameOne/files/4088667/theme.zip)

[ddyer0]

If you run the above example under xcode and the mac's simulator, it locks up pretty quickly

[shannah]

Does it make a difference if you call cleanup on your Media objects when you're done with them?

[ddyer0]

calling cleanup helps a lot, but I don't think it fixes the problem. Running on the simulator locks up after a while instead of immediately. Running on real hardware hasn't failed yet, but I'm pretty sure it will eventually.

Also, just for the record, this appears to be a complex interaction between the GC and the native sound code. My actual app fails in seconds, so the length of time this simple test runs isn't very comforting.

[ddyer0]

with cleanup my Ipad crashed after an hour.