search

edusperoni / nativescript-mqtt

MQTT 3.1.1 for Nativescript
Apache License 2.0
12 stars 3 forks source link

Connection Lost after a minute #9

Open karthikbalu opened 5 years ago

karthikbalu commented 5 years ago

Connection is getting lost after few minutes of active, I tried reinitialising the mqttclient but continously connecting and lost connection as below

'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...'

Which platform(s) does your issue occur on?

Please, provide the following version numbers that your issue occurs with:

Please, tell us how to recreate the issue in as much detail as possible.

It is a simple project that connects to EMQ MQTT broker, the connection works but doesnot stay connected no longer than a minute time and even reconnecting it constantly fails

Is there any code involved?

const MQTTCommon = require("nativescript-mqtt/common");
var gencode = function () {
    function s4() {
        return Math.floor((1 + Math.random()) * 0x10000)
            .toString(16)
            .substring(1);
    }
    return s4() + s4() + '-' + s4() + '-' + s4() + '-' + s4() + '-' + s4() + s4() + s4();
}

module.exports = {
    config: {
        mqtt_client: null,
        surprise: false,
        mqtt_username: "",
        mqtt_password: "",
        topics: [],
        mqtt_clientOptions: {
            host: "192.168.1.19",
            port: 8083,
            clientId: gencode(),
            path: '/mqtt',
            useSSL: false
        }
    },
    init: function (callback) {

        console.log('initialized connection');
        this.mqtt_clientOptions = this.config.mqtt_clientOptions;

        let self = this;

        this.mqtt_client = new MQTTClient(this.mqtt_clientOptions);

        this.mqtt_client.onConnectionFailure.on(function (err) {
            console.log("Connection failed: " + err);
            self.init();
        });

        this.mqtt_client.onConnectionSuccess.on(function () {
            console.log("Connection success!");
            if (callback) callback();
            else self.resubscribe();
        });

        this.mqtt_client.onConnectionLost.on(function (err) {
            console.log("Connection lost: " + err);
            self.init();
        });

        // this.mqtt_client.onMessageArrived.on(function (message) {
        //     console.log("Message received: " + JSON.parse(message.payload).message);
        // });

        console.log("mqtt init called");
        this.connect();
    },
    connect: function () {
        console.log("connecting...");
        try {
            this.mqtt_client.connect(this.config.mqtt_username, this.config.mqtt_password);
        }
        catch (e) {
            console.log("Caught error: " + e);
        }
    },
    getClient: function () {
        return this.mqtt_client;
    },
    subscribe: function (topic) {
        const self = this;
        if (self.mqtt_client.connected) {
            console.log('subscribe to ' + topic);
            if (self.config.topics.indexOf(topic) == -1) {
                try {
                    self.config.topics.push(topic);
                    this.mqtt_client.subscribe(topic);
                }
                catch (e) {
                    console.log("Caught error: " + e);
                }
            }
        }
    },
    resubscribe: function () {
        const self = this;
        if (self.mqtt_client.connected) {
            console.log('re-subscribing all');
            for (var i = 0; i < this.config.topics.length; i++) {
                var element = this.config.topics[i];
                try {
                    this.mqtt_client.subscribe(element);
                    console.log('resubscribed to ', element);
                }
                catch (e) {
                    console.log("Caught error: " + e);
                }
            }
        }
    },
    publish: function (message) {
        console.log("message", message);
        // var mqtt_message = new MQTTCommon.Message(message);
        try {
            console.log('message published to ' + message.topic);
            this.mqtt_client.publish(message);
        }
        catch (e) {
            console.log("Caught error: " + e);
        }
    }
}
DarrellBrogdon commented 5 years ago

I'm seeing this behavior as well.

edusperoni commented 5 years ago

Do subsequent connections fail after a minute or instantly?

DarrellBrogdon commented 5 years ago

Subsequent connections fail as well. For me, I'm also seeing this in my broker logs:

1551832536: Client my-test-client has exceeded timeout, disconnecting.
1551832536: Socket error on client my-test-client, disconnecting.
edusperoni commented 5 years ago

Do they fail after a minute or instantly?

DarrellBrogdon commented 5 years ago

They fail after what appears to be about 2.5 minutes.

Here is the complete broker log from the time the client connects to the time it disconnects. "my-test-client" is the client ID of my NativeScript app.

1551836600: New client connected from ::ffff:172.17.0.1 as my-test-client (c1, k60, u'').
1551836600: No will message specified.
1551836600: Sending CONNACK to my-test-client (0, 0)
1551836600: Received SUBSCRIBE from my-test-client
1551836600:     /device/test-device-id-1/updated (QoS 0)
1551836600: my-test-client 0 /device/test-device-id-1/updated
1551836600: Sending SUBACK to my-test-client
1551836605: Received PINGREQ from mqttjs_a77046d5
1551836605: Sending PINGRESP to mqttjs_a77046d5
1551836607: Received PINGREQ from mqttjs_8227a3cb
1551836607: Sending PINGRESP to mqttjs_8227a3cb
1551836615: Received PINGREQ from mosqsub|6138-Darrells-M
1551836615: Sending PINGRESP to mosqsub|6138-Darrells-M
1551836630: Received PINGREQ from mqttjs_f4fd3342
1551836630: Sending PINGRESP to mqttjs_f4fd3342
1551836658: Received PINGREQ from mqttjs_78ec9a9d
1551836658: Sending PINGRESP to mqttjs_78ec9a9d
1551836660: Received PINGREQ from my-test-client
1551836660: Sending PINGRESP to my-test-client
1551836665: Received PINGREQ from mqttjs_a77046d5
1551836665: Sending PINGRESP to mqttjs_a77046d5
1551836667: Received PINGREQ from mqttjs_8227a3cb
1551836667: Sending PINGRESP to mqttjs_8227a3cb
1551836675: Received PINGREQ from mosqsub|6138-Darrells-M
1551836675: Sending PINGRESP to mosqsub|6138-Darrells-M
1551836690: Received PINGREQ from mqttjs_f4fd3342
1551836690: Sending PINGRESP to mqttjs_f4fd3342
1551836718: Received PINGREQ from mqttjs_78ec9a9d
1551836718: Sending PINGRESP to mqttjs_78ec9a9d
1551836725: Received PINGREQ from mqttjs_a77046d5
1551836725: Sending PINGRESP to mqttjs_a77046d5
1551836727: Received PINGREQ from mqttjs_8227a3cb
1551836727: Sending PINGRESP to mqttjs_8227a3cb
1551836735: Received PINGREQ from mosqsub|6138-Darrells-M
1551836735: Sending PINGRESP to mosqsub|6138-Darrells-M
1551836750: Received PINGREQ from mqttjs_f4fd3342
1551836750: Sending PINGRESP to mqttjs_f4fd3342
1551836751: Client my-test-client has exceeded timeout, disconnecting.
1551836751: Socket error on client my-test-client, disconnecting.
edusperoni commented 5 years ago

I'll update the plugin this week to expose more options from paho-mqtt, including a trace option to better debug this issue. This could be an issue with the keepalive, which is also not being exposed.

For the time being, you can try reconnecting whenever the connection stops (which is what I'm doing)

DarrellBrogdon commented 5 years ago

I actually already exposed keepAliveInterval in my testing. I also set up calls to start/get/stop traceLog with a reduced keepAliveInterval of 5. Here is the output of getTraceLog() if its any help:

CONSOLE LOG file:///app/status-view-model.js:56:16: Client.startTrace
CONSOLE LOG file:///app/status-view-model.js:56:16:   "2019-03-06T02:13:11.338Z"
CONSOLE LOG file:///app/status-view-model.js:56:16:   "@VERSION@-@BUILDLEVEL@"
CONSOLE LOG file:///app/status-view-model.js:56:16: Client.connect
CONSOLE LOG file:///app/status-view-model.js:56:16:   {"userName":"","password":"******","useSSL":false,"keepAliveInterval":5,"mqttVersionExplicit":false,"mqttVersion":4,"cleanSession":true}
CONSOLE LOG file:///app/status-view-model.js:56:16:   null
CONSOLE LOG file:///app/status-view-model.js:56:16:   false
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._socket_send
CONSOLE LOG file:///app/status-view-model.js:56:16:   {"type":1,"userName":"","password":"******","useSSL":false,"keepAliveInterval":5,"mqttVersionExplicit":false,"mqttVersion":4,"cleanSession":true,"clientId":"my-test-client"}
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._on_socket_message
CONSOLE LOG file:///app/status-view-model.js:56:16:   {}
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._handleMessage
CONSOLE LOG file:///app/status-view-model.js:56:16:   {"type":2,"returnCode":0}
CONSOLE LOG file:///app/status-view-model.js:56:16: Client.subscribe
CONSOLE LOG file:///app/status-view-model.js:56:16:   "/device/test-device-id-1/updated"
CONSOLE LOG file:///app/status-view-model.js:56:16:   {}
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._socket_send
CONSOLE LOG file:///app/status-view-model.js:56:16:   {"type":8,"topics":["/device/test-device-id-1/updated"],"requestedQos":[0],"messageIdentifier":1}
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._on_socket_message
CONSOLE LOG file:///app/status-view-model.js:56:16:   {}
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._handleMessage
CONSOLE LOG file:///app/status-view-model.js:56:16:   {"type":9,"messageIdentifier":1,"returnCode":{"0":0}}
CONSOLE LOG file:///app/status-view-model.js:56:16: Pinger.doPing
CONSOLE LOG file:///app/status-view-model.js:56:16:   "send PINGREQ"
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._on_socket_message
CONSOLE LOG file:///app/status-view-model.js:56:16:   {}
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._handleMessage
CONSOLE LOG file:///app/status-view-model.js:56:16:   {"type":13}
CONSOLE LOG file:///app/status-view-model.js:56:16: Pinger.doPing
CONSOLE LOG file:///app/status-view-model.js:56:16:   "send PINGREQ"
CONSOLE LOG file:///app/status-view-model.js:56:16: Client._disconnected
CONSOLE LOG file:///app/status-view-model.js:56:16:   8
CONSOLE LOG file:///app/status-view-model.js:56:16:   "AMQJS0008I Socket closed."
CONSOLE LOG file:///app/status-view-model.js:56:16: Client.getTraceLog
CONSOLE LOG file:///app/status-view-model.js:56:16:   "2019-03-06T02:13:24.299Z"
CONSOLE LOG file:///app/status-view-model.js:56:16: Client.getTraceLog in flight messages
CONSOLE LOG file:///app/status-view-model.js:56:16: undefined
edusperoni commented 5 years ago

Could you also try it using https://www.eclipse.org/paho/clients/js/utility/ ?

So we can check if the problem is with paho mqtt or with NS websockets. If it also disconnects on the web, it might be an issue with the web server

Edit: Untick "automatic reconnect"

DarrellBrogdon commented 5 years ago

I get an immediate connection timeout when I visit that page:

Wed, 06 Mar 2019 23:19:27 GMT - INFO - Starting Eclipse Paho JavaScript Utility.
Wed, 06 Mar 2019 23:19:37 GMT - INFO - Connecting to Server: [Host: iot.eclipse.org, Port: 443, Path: /ws, ID: js-utility-Eyicv]
Wed, 06 Mar 2019 23:19:43 GMT - ERROR - Failed to connect. [Error Message: AMQJSC0001E Connect timed out.]
edusperoni commented 5 years ago

@DarrellBrogdon use your own broker, I think iot.eclipse.org isn't working atm

edusperoni commented 5 years ago

I've exposed most of the stuff in a separate branch, see https://github.com/edusperoni/nativescript-mqtt/pull/10.

The new implementation properly exposes trace options and allows you to use most of the options given by the Paho MQTT Client. This is not final, but probably somewhat close to it.

karthikbalu commented 5 years ago

They fail after what appears to be about 2.5 minutes.

Here is the complete broker log from the time the client connects to the time it disconnects. "my-test-client" is the client ID of my NativeScript app.

1551836600: New client connected from ::ffff:172.17.0.1 as my-test-client (c1, k60, u'').
1551836600: No will message specified.
1551836600: Sending CONNACK to my-test-client (0, 0)
1551836600: Received SUBSCRIBE from my-test-client
1551836600:   /device/test-device-id-1/updated (QoS 0)
1551836600: my-test-client 0 /device/test-device-id-1/updated
1551836600: Sending SUBACK to my-test-client
1551836605: Received PINGREQ from mqttjs_a77046d5
1551836605: Sending PINGRESP to mqttjs_a77046d5
1551836607: Received PINGREQ from mqttjs_8227a3cb
1551836607: Sending PINGRESP to mqttjs_8227a3cb
1551836615: Received PINGREQ from mosqsub|6138-Darrells-M
1551836615: Sending PINGRESP to mosqsub|6138-Darrells-M
1551836630: Received PINGREQ from mqttjs_f4fd3342
1551836630: Sending PINGRESP to mqttjs_f4fd3342
1551836658: Received PINGREQ from mqttjs_78ec9a9d
1551836658: Sending PINGRESP to mqttjs_78ec9a9d
1551836660: Received PINGREQ from my-test-client
1551836660: Sending PINGRESP to my-test-client
1551836665: Received PINGREQ from mqttjs_a77046d5
1551836665: Sending PINGRESP to mqttjs_a77046d5
1551836667: Received PINGREQ from mqttjs_8227a3cb
1551836667: Sending PINGRESP to mqttjs_8227a3cb
1551836675: Received PINGREQ from mosqsub|6138-Darrells-M
1551836675: Sending PINGRESP to mosqsub|6138-Darrells-M
1551836690: Received PINGREQ from mqttjs_f4fd3342
1551836690: Sending PINGRESP to mqttjs_f4fd3342
1551836718: Received PINGREQ from mqttjs_78ec9a9d
1551836718: Sending PINGRESP to mqttjs_78ec9a9d
1551836725: Received PINGREQ from mqttjs_a77046d5
1551836725: Sending PINGRESP to mqttjs_a77046d5
1551836727: Received PINGREQ from mqttjs_8227a3cb
1551836727: Sending PINGRESP to mqttjs_8227a3cb
1551836735: Received PINGREQ from mosqsub|6138-Darrells-M
1551836735: Sending PINGRESP to mosqsub|6138-Darrells-M
1551836750: Received PINGREQ from mqttjs_f4fd3342
1551836750: Sending PINGRESP to mqttjs_f4fd3342
1551836751: Client my-test-client has exceeded timeout, disconnecting.
1551836751: Socket error on client my-test-client, disconnecting.

For me too, when it fails after a minute, subsequent reconnections also fails when i try to reconnect on failure and loops infinitely...

Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...' JS: 'Connection success!' JS: 'Connection lost: AMQJS0008I Socket closed.' JS: 'initialized connection' JS: 'mqtt init called' JS: 'connecting...'

edusperoni commented 5 years ago

Have you had any success testing on https://www.eclipse.org/paho/clients/js/utility/ ? Does it also fail there (uncheck auto reconnect, use the same broker you're trying to use)

DarrellBrogdon commented 5 years ago

I haven't had a chance to test it yet. I'll try to do so today.

edusperoni commented 5 years ago

I can't reproduce it using broker.mqttdashboard.com. I'm testing the version 2.0 today and spent a good deal of time today connected to it while coding and writing tests. Also, I added a reconnect button to the demo, and it never failed to reconnect.

I've pushed some code to the paho-refactor branch if you want to run the demo with the reconnect button+time to connect message

Edit:

last test result:

image

The second mqtt connect was me hitting the reconnect button

DarrellBrogdon commented 5 years ago

This is what I see:

CONSOLE LOG file:///app/main-view-model.js:126:20: connecting
CONSOLE LOG file:///app/main-view-model.js:126:20: Mqtt connected
CONSOLE LOG file:///app/main-view-model.js:126:20: Mqtt connection stablished {"reconnect":false,"uri":"ws://broker.mqttdashboard.com:8000/mqtt"}
CONSOLE LOG file:///app/main-view-model.js:126:20: Message received. Topic: testtopic/1. Payload: iozlukjhgf
CONSOLE LOG file:///app/main-view-model.js:126:20: Mqtt connection lost: {"errorCode":4,"errorMessage":"AMQJS0004E Ping timed out.","reconnect":false,"uri":"ws://broker.mqttdashboard.com:8000/mqtt"}
CONSOLE LOG file:///app/main-view-model.js:73:24: Time connected: 3m 0.3739999999999952s
edusperoni commented 5 years ago

As I said before, please try using https://www.eclipse.org/paho/clients/js/utility/ to check if the error is exclusive to mobile or may be an issue with your broker...

edusperoni commented 5 years ago

Ok, I see that you used broker.mqttdashboard.com, so let's establish some things:

  1. If your connection fails, it's expected that MQTT also fails and disconnects. This is expected behavior.
  2. If your MQTT disconnects, you have to deal with it. This called fault tolerance and recovery and every software in existence has to implement it in some kind of way. In this case, you should probably try to reconnect.

Now let's take a look at this issue. If you got a disconnect, it may be because your connection failed (1). So you try recovering it (2). If you're unable to reconnect, but can reconnect when restarting the app, then we have an issue.

If you get CONSTANTS disconnects after a certain amount of time, this might be an issue with this plugin. But if it works on other brokers (like mqttdashboard) and not yours, might be an issue with YOUR broker.

So far, this is the results of my testing:

  1. I've experienced disconnects only on large amounts of time using broker.mqttdashboard.com, this was one disconnect after 20 minutes and no disconnects for over an hour.
  2. every disconnect I experienced was recoverable by reconnecting

This issue is about two things:

  1. disconnect after a minute
  2. problems reconnecting

Unless you can provide me with an example in which 1 or 2 happens, I'm going to have to close this issue.

DarrellBrogdon commented 5 years ago

In my project (which uses the master branch) I was able to reconnect, but it does disconnect after about a minute regardless.

I'll test the branch with connecting to my broker to see what happens. I feel like it disconnecting after 20 minutes is ok but 1-3 minutes is problematic. That seems really inefficient to have to reconnect that much.

DarrellBrogdon commented 5 years ago

Also, when you say try using https://www.eclipse.org/paho/clients/js/utility/, what values should I be using in the form?

edusperoni commented 5 years ago

Host: your broker (192.168.etc or my.broker.com or broker.mqttdashboard.com) Port: port you're using to connect Path: /mqtt /ws etc (or nothing, based on your broker)

Check/Uncheck TLS accordingly (same as useSSL). Untick "Automatic Reconnect".

If you uncheck TLS and hit connect, chrome may complain about insecure stuff, so you have to click the little shield by the address bar and click "load unsafe scripts".

My best guess is that your network is failing sometimes and dropping some MQTT packets that trigger a disconnect.

DarrellBrogdon commented 5 years ago

My broker is running locally on my machine for now so I'm not able to use that utility to test it.

If I try to connect to broker.mqttdashboard.com I get "Failed to connect: AMQJS0007E Socket error:undefined." Port=8000, Client ID=js-utility-VEdxd, Path=/mqtt, TLS and Automatic Reconnect are unchecked.

edusperoni commented 5 years ago

You have to check "load unsafe scripts" by the address bar if not using SSL.

DarrellBrogdon commented 5 years ago

It's using SSL. I don't have any VPN or filtering running that would affect this so I'm not sure why it's unable to connect.

DarrellBrogdon commented 5 years ago

So I put my broker on a remote server and tested it both by using mosquitto_sub on the command line and using http://www.hivemq.com/demos/websocket-client/ (I still can't get https://www.eclipse.org/paho/clients/js/utility/ working). Neither of those time out. However, when I run the latest demo code I still get the "AMQJS0008I Socket closed." error.

CONSOLE LOG file:///app/main-view-model.js:187:20: Connecting to ws://my-broker-host:9001
CONSOLE LOG file:///app/main-view-model.js:187:20: onConnectionSuccess
CONSOLE LOG file:///app/main-view-model.js:187:20: onConnected: {"reconnect":false,"uri":"ws://my-broker-host:9001"}
CONSOLE LOG file:///app/main-view-model.js:187:20: onSubscribeSuccess: {"grantedQos":0}
CONSOLE LOG file:///app/main-view-model.js:187:20: (Promise) Subscribed to /device/test-device-id-1/updated {"grantedQos":0}
CONSOLE LOG file:///app/main-view-model.js:187:20: onMessageArrived: {"destinationName":"/device/test-device-id-1/updated","payloadString":"Test Message 5","duplicate":false,"retained":false,"qos":0}
CONSOLE LOG file:///app/main-view-model.js:187:20: onConnectionLost: {"errorCode":8,"errorMessage":"AMQJS0008I Socket closed.","reconnect":false,"uri":"ws://my-broker-host:9001"}
CONSOLE LOG file:///app/main-view-model.js:104:24: Time connected: 2m 37.07599999999999s
edusperoni commented 5 years ago

what android version are you using? are you using the android emulator, physical device or other emulators (like genymotion)?

DarrellBrogdon commented 5 years ago

I'm using iOS actually. I'm using the iPhone XR simulator running iOS 12.1.

edusperoni commented 5 years ago

Could you try running on Android so we can determine if it's something specific to iOS?

DarrellBrogdon commented 5 years ago

Looks like you may be right. It has been running for about 10 minutes now on the Android emulator (Pixel_2_XL_Pie avd) without any disconnect.

edusperoni commented 5 years ago

Seems an issue with websockets on iOS... https://github.com/NathanaelA/nativescript-websockets/issues/61 https://github.com/NathanaelA/nativescript-websockets/issues/63

edusperoni commented 5 years ago

@DarrellBrogdon can you test something out?

Try setting application.ios.nativeApp.idleTimerDisabled = true; (application comes from tns-core-modules/application).

Maybe the issue comes from app inactivity, as it seems to close sockets

DarrellBrogdon commented 5 years ago

I'm getting this error when I do: file:///app/app.js:8:26: JS ERROR TypeError: null is not an object (evaluating 'application.ios.nativeApp.idleTimerDisabled = true')

edusperoni commented 5 years ago

Try this:

declare var UIApplication: any;

UIApplication.sharedApplication.idleTimerDisabled = true;

if that doesn't work, just create a dummy button and tap it from time to time, just to test if the socket closing is a result from inactivity

DarrellBrogdon commented 5 years ago

I'm using Vanilla JS but testing with a dummy button, clicking every ~5-10 seconds didn't seem to make a difference. It still eventually loses the connection.

grantwparks commented 4 years ago

Any more solution ideas?